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EXECUTIVE SUMMARY 



The evaluation of the New Century High Schools (NCHS) initiative examined operations 
and student outcomes in 75 schools from 2002-03 through 2005-06. This report, the final in a 
series of annual evaluation reports, presents data collected over those years, with a focus on 
school year 2005-06. 

The NCHS initiative grew out of a program theory that emphasized small school 
enrollments, instructional rigor and focus, youth development and positive relationships, and the 
mobilization of supplementary educational resources. The initiative’s central method for 
promoting these school features was through frameworks and supports for disciplined 
innovation, which relies on informed problem-solving to address student needs within broad, 
agreed-upon boundaries and which emphasizes continuous data-driven feedback and tailored 
assistance. 

The evaluation used the percent of students in the Class of 2006 who graduated on time 
as its prime indicator of success. The study’s central finding is that NCHS schools graduated 
more students on time than did larger New York City schools with comparable youngsters, by 18 
percentage points (78.2 percent versus 60.6 percent), and more students than did New York City 
high schools generally, by 20 percentage points (78.2 percent versus 58.2 percent). 

NCHS effects were notable with regard to drop-out prevention and on-time graduation. 
About 17 percent of students in comparison-group schools dropped out and about 22 percent 
remained enrolled in high school for a fifth year, compared to 3 percent and 19 percent, 
respectively, in NCHS schools. Examination of the Class of 2006 graduates in the two groups of 
schools indicates that graduates of comparison-group schools were more likely to earn a Regents 
diploma or Advanced Regents diploma, however, than were NCHS graduates (67 percent versus 
46 percent). When the unit of comparison is students rather than graduates, however, the 
difference is less stark, with 41 percent of comparison-group students and 36 percent of NCHS 
students earning a Regents or Advanced Regents diploma. NCHS students were slightly more 
likely to earn a Regents or Advanced Regents diploma than were New York city students in the 
class of 2005 (36 percent, compared to 35 percent). 

Compared to New York City high schools generally, NCHS schools enrolled students 
from backgrounds characterized by higher rates of poverty and lower eighth-grade test scores. 
While in high school, however, NCHS students surpassed city wide averages in school attendance 
and grade promotion, although their suspension rates were higher. The average daily attendance 
of NCHS students in 2005-06 was 84 percent, compared to 81 percent for New York City high 
school students overall (including students in ungraded special education). Median NCHS 
student attendance was 91 percent. NCHS ninth-graders were promoted to the next grade at a 
rate of 80 percent, compared to 72 percent city wide (based on 2004-05 data for the city). NCHS 
students were suspended from school at a rate of almost 8 percent, however, compared to a 
city wide high school rate of 6.5 percent in 2006. (NCHS suspension rates may be the result of 
the schools’ adoption of strict standards and not necessarily a reflection of relatively poorer 
student behavior.) 
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NCHS classes following the Class of 2006 differ from their 2006 predecessors in lower 
rates of high school attendance and credit accrual, higher rates of school suspension, and higher 
Regents passing rates. This pattern is consistent with the eighth-grade profile of these 
subsequent classes. As eighth-graders, later NCHS entrants had poorer attendance and higher 
suspension rates, comparable reading scores, and higher math scores. 

Evaluators used both hierarchical multivariate statistical methods and qualitative 
approaches to identify school features associated with student outcomes. Using the statistical 
methods available to the evaluation, the most important school-level influence on student 
performance, as measured by credit accrual, was a construct evaluators labeled “the quality of 
instructional systems.” The index defining this construct included measures of the perceived 
alignment of instruction with Regents standards, agreement on educational focus, the 
effectiveness of principal leadership, the quality and amount of professional development, 
teacher influence, and professional collaboration on instruction. Evaluators measured the effects 
on credit accumulation of higher scores on the index of instructional systems quality. The 
maximum hypothetical effect was found to be seven credits over a two-year period, although no 
NCHS school matched the characteristics of the extreme conditions at either end of this 
spectrum. Practically speaking, a typical student’s credit accumulation was 1.4 credits higher in 
a school with a better score on the index (by one standard deviation), compared to 1.4 credits 
lower in a school with a worse score on the index (by one standard deviation). 

Case studies in the 2005-06 school year and earlier evaluation findings illuminated the 
influence on student outcomes of conditions that were fairly uniform across NCHS schools. 
Influential factors included small enrollments, close student- teacher relationships and adult 
mentoring of youth, the extension of student learning outside the regular school setting and 
school day, and the use of data to track student performance. 

Using surveys of teachers, principals, and students to assess educational conditions in 
each successive wave of new NCHS schools, evaluators found that perceptions of most measured 
conditions remained stable over the four years of the evaluation. The perceived alignment of 
instruction with Regents requirements, however, rose over time, according to surveys of 
teachers. And perceptions of the quality of student discipline and teacher influence on policy 
and curriculum declined, according to teacher survey responses. 

Perceptions of school conditions that were both positive and stable included the 
following: teachers’ expectations for students, teacher-student relationships, agreement on the 
schools’ educational focus, the effectiveness of principal leadership, professional collaboration, 
quality of teacher-made tests, and availability of supplementary learning opportunities and of 
instructional materials. Perceptions of school conditions that were both somewhat negative and 
stable included the following: the adequacy of school facilities, the quality and amount of 
professional development, and parental involvement. 
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1. BACKGROUND ON THE NEW CENTURY HIGH SCHOOLS 

AND THIS REPORT 



Between September 2002 and September 2005, the New York City Department of 
Education (DOE) opened 162 new small secondary schools, tripling the number of public 
secondary institutions in the City. Working closely with the DOE, the United Eederation of 
Teachers, and the Council of Supervisors and Administrators, New Visions for Public Schools 
provided leadership and technical support for 75 of these startups, known as the New Century 
High Schools (NCHS schools). The Bill & Melinda Gates Eoundation, the Carnegie Corporation 
of New York, and the Open Society Institute provided supplementary financial support for the 
planning and initial implementation of NCHS schools. 

In 2002, New Visions for Public Schools asked Policy Studies Associates, Inc., to 
evaluate the schools it planned to open between September 2002 and September 2005. This is 
the last in a series of reports describing conditions and achievement outcomes in those 75 
schools. The report summarizes research efforts over four years, with a focus on school year 
2005-06. It relates youth outcomes to students’ background characteristics and to the 
characteristics of their schools, focusing on school characteristics that New Visions believed 
would create educational value. 



Program Theory Underlying the NCHS Initiative 

Exhibit 1 presents New Visions’ conception of the process by which the NCHS initiative 
would create and operate successful small schools of about 400 students each. To support efforts 
at the school level. New Visions offered school developers small planning grants and 
supplemental operating grants of $400,000 over four years. To fuel collaborative efforts, the 
operating grants were provided to private nonprofit organizations, working in collaboration with 
professional educators, for work within or on behalf of individual schools. In addition. New 
Visions provided direct technical assistance to NCHS schools, and it cultivated systemic support 
within the DOE, the local DOE regions, and labor organizations. 

The preceding steps constituted the initiative’s process theory (or ideas about how to 
motivate positive change). New Visions expressed their intervention theory (or ideas about how 
good schools would function) in the following ten principles: 

■ Rigorous instructional program, enabling every student to master challenging 
skills, content knowledge, and state standards through relevant, individualized, in- 
depth, and inquiry-based teaching 

■ Personalized relationships between students and teachers, characterized by close 
continuous communication and each student having at least one adult to 
coordinate the support needed for the student to achieve educational goals 
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Exhibit 1 
Program Theory 



INITIAL PHASE 



LATER PHASE 



ACTION STEPS 




SHORT-TERM 

OUTCOMES 


■ 


INTERMEDIATE 

OUTCOMES 




LONG-TERM 

OUTCOMES 



Establish a grant- 
making, planning, 
and development 
process 



Provide direct 
support to grantees 
in developing 
effective new 
schools 



Involve community 
partners in planning 
and operating new 
schools 



Cultivate systemic 
support within DOE, 
regions, and unions 





Establish new small 






high schools 




V 




J 



New Century High Schools 
provide: 



Rigorous instructional 
program 

Personalized relationships 
between adults and 
students 

A clear focus and high 
expectations for students 

Instructional leadership 
focused on student 
achievement 

School-based professional 
development and 
collaboration 

Meaningful assessment of 
student learning 

Partnerships with 
community organizations 

Family partnership and 
involvement 

Youth participation and 
development 

Effective uses of 
technology and resources 



Schools attract 
cross-section of 
the student 
population in 
communities they 
serve 



Enrolled 
students 
outperform 
students with 
similar 

educational and 
personal 
backgrounds 



Students are 
positively 
engaged with 
their school and 
community, and 
prepared for 
postsecondary 
experiences 



Systemic adoption of New 
Century elements across 
New York City high 
schools 



Improved quality of 
learning experiences for 
youth, especially those 
from most disadvantaged 
communities 
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■ Clear focus on teaching and learning and corresponding expectations that every 
student will succeed 

■ Instructional leadership through effective collaboration and school- wide support 
for teaching and learning 

■ School-based teacher-driven professional development and collaboration that is 
results-driven, standards-based, and embedded in the daily work of the school 

■ Meaningful continuous assessment to diagnose student needs and improve 
instruction 

■ Community partners that offer support and opportunities for students, families, 
and the school community and contribute significantly to the school’s planning 
processes, governance, and operations 

■ Family/caregiver partnership and involvement in governance and the design of 
the school’s education program 

■ Youth participation and development, characterized by student voice in teaching 
and learning and an educational focus on the development of students’ social and 
emotional skills 

■ Effective uses of technology and other resources, including print, visual, audio, 
and electronic resources to facilitate learning and school operations 

New Visions characterized its approach to educational improvement as disciplined 
innovation, in which the 10 NCHS principles provided a framework and direction for 
improvement efforts that could incorporate school themes, branded education-reform approaches 
(e.g., America’s Choice, Expeditionary Learning), and other organizational and curricular 
approaches. Disciplined innovation, as employed by New Visions and others, promotes 
informed problem-solving to address student needs within broad, agreed-upon boundaries and 
with the use of continuous data-driven feedback and tailored supports. 

New Visions also established numerical benchmarks intended to serve as aspirational 
goals for NCHS schools. The benchmarks encouraged every NCHS school to graduate at least 
80 percent of its students in four years and to maintain an attendance rate of at least 92 percent. 
These benchmarks were intended to motivate educators and other stakeholders to assess 
students’ progress toward graduation on a continuing basis and to take actions as needed to make 
sure that students stayed on course for on-time graduation. 



Organization of Report 

This report relates achievement outcomes in NCHS schools to the program model 
described above. The report is presented in nine chapters and two appendixes. 
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Chapter 1 describes the initiative’s program theory. Chapter 2 reviews prior findings and 
presents an overview of the design for the 2005-06 research. Methods are presented in detail in 
later chapters. Chapter 3 describes students attending NCHS schools in 2005-06. Chapter 4 
presents stakeholders’ assessments of the schools’ learning environments over four years. 

Chapter 5 presents graduation rates for the NCHS Class of 2006 and compares NCHS 
results with citywide outcomes and with outcomes in comparison-group schools. Chapter 6 
examines the progress of NCHS students scheduled to graduate in later classes. 

Chapters 7 and 8 present quantitative and qualitative findings regarding relationships 
among NCHS inputs, processes, and outcomes. Chapter 7 presents findings from multivariate 
quantitative analyses. Chapter 8 presents findings derived from case studies over four years. 
Chapter 9 presents our conclusions. Appendix A provides descriptive statistics for survey data 
summarized in Chapter 4. Appendix B presents comparisons of the NCHS and contrast-group 
students at entry to high school. 
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2. REVIEW OF EARLIER FINDINGS AND 
RESEARCH DIRECTIONS 



Findings through 2004-05 

Findings regarding the NCHS initiative through 2004-05 were very promising. More 
than three-quarters of students in NCHS schools for three years attained credits and passed 
Regents exams at rates congruent with on-time graduation. And in all years, NCHS students out- 
performed youngsters with whom they were demographically and academically matched based 
on middle school performance. Qualitative case study data provided compelling evidence of the 
utility of specific NCHS process strategies. 

■ Collaboration among New Visions and powerful city-level stakeholders — 
foundation officers, labor leaders, school executives, and the mayor — enabled 
coherent change at an aggressive pace and on a large scale. The centerpiece of 
the NCHS reform in its first years was, almost certainly, the collaborative 
enterprise itself. It ensured the schools would have needed political and financial 
support. 

■ School-level collaboration and partnerships varied in their longevity and 
effectiveness. Some partnerships developed out of deeply compatible interests 
and were integral to school life; others may have been useful in the early 
formation and establishment of the schools but were not sustained over the long 
term. 

■ Planning helped prospective principals to imagine rigorous instructional programs 
with real-world relevance. But successful program implementation required 
individuals with the means to attract, manage, and develop teacher talent. It also 
required leaders with the skills and knowledge to marshal other resources from 
many sources. 

■ Smaller enrollments facilitated social relationships among students and helped 
teachers in their efforts to personalize instruction. 

■ Professional development worked best, as theorized, when it was school-initiated 
and school-based, but regionally based professional-development sessions 
remained part of the school system’s repertoire. 



Research in 2005-06 

Some reformers advocate approaches to school development that are likely to provide 
students with customized learning experiences (Cuban & Shipps, 2000). Others prefer 
approaches that substantially regulate teacher and student behavior (Ross, Hogaboam-Gray, & 



5 




Hannay, 2001). NCHS reformers were in the first camp. They encouraged schools to take 
initiative and to be entrepreneurial in responding to opportunities and in solving problems. New 
Visions relied on schools to determine ways to achieve the personalization and engagement 
needed for positive educational performance. They supported schools, in part, through the 
sharing of data and analyses that allowed schools to track their students’ progress to graduation. 
Their methodology of disciplined innovation allowed educators to devise specialized solutions 
while also engaged in continuous assessment and efforts toward the 80/92 benchmarks. 

The question for study in 2005-06, Year 4 of the initiative and of the evaluation, was 
whether this change paradigm worked. Would it be possible with these methods to push 
anticipated four-year high school graduation rates in NCHS schools from less than 60 percent, as 
was the norm in New York City schools, to about 80 percent, as New Visions aimed? Below is 
an overview of the study’s design and major hypotheses. 



Research Design 

In Year 4, as before, we employed a mixed- model research design, an approach that 
combines both quantitative and qualitative methods (Tashakkori & Teddlie, 1998). The 
advantage of this strategy was that it allowed us to wrestle with problems flexibly using methods 
that had differing strengths and weaknesses (Brewer & Hunter, 1989). 

In assessing achievement outcomes, we used a quasi-experimental design. This approach 
had the virtue of inclusiveness as well as relative objectivity. Achievement data were available 
for almost all NCHS students, and survey data describing school conditions were available for 
almost all NCHS schools (68 of 75). 

A predictable problem in using quantitative data to understand school processes is the 
likely insensitivity of fixed-response survey items to subtle organizational conditions and 
interrelationships. To invite the unexpected into our data set, we also looked at school processes 
in a holistic fashion (Yin, 1994) in a few settings (N = 6) using student records, repeated school- 
level interviews, and structured classroom observations as data sources. By triangulating data 
within case study schools and replicating analyses across schools, we hoped both to validate 
qualitative findings and to be in a position to elaborate on quantitative findings. 



Hypotheses 

Given earlier findings regarding NCHS students progress over time, it seemed unlikely 
that the schools would fully achieve New Visions’ aims of an 80 percent four-year graduation 
rate without modification to the NCHS model and school practice. We anticipated that, without 
such modifications, the initiative would come closest to achieving its graduation goals with the 
Class of 2006. 

We had a rich literature from which to draw hypotheses about factors influencing 
academic outcomes. We considered four sets of explanations: student characteristics, staff 
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characteristics, attention to instruction, and arrangements that bolster students emotionally (e.g., 
small enrollments, advisories, etc.). These frameworks are thoroughly discussed in the literature 
(e.g., Borman & Rachuba, 2001; Phillips, 1997) and, in the case of attention to instruction and 
emotional support, well aligned with NCHS core principles.^ Controlling for student individual 
and aggregate characteristics and for staff aggregate characteristics, we wagered that credit 
accumulation would be higher in NCHS schools that were attentive to instruction and also 
emotionally supportive. 



Preview of Year 4 Findings 

The lack of variation in staff characteristics and school supportiveness — most principals 
and teachers were inexperienced, and most schools were small and supportive — constrained the 
factors we were able to investigate with statistical precision. Student characteristics and the 
quality of instructional processes did, however, exert a statistically significant influence on 
achievement outcomes. Qualitative evidence suggests that student outcomes were also 
influenced by small enrollments, close student-teacher relationships and adult mentoring of 
youth, the availability of supplemental resources to extend instructional time, and the use of data 
to track student performance. 



* Researchers have associated a number of factors, such as “firm leadership” and “high expectations,” with 
“effective schooling.” Rowe, Hill, and Holmes-Smith (1995) discounted much of this work, however, on 
methodological grounds. They concluded that most school-reform initiatives were poorly supported by evidence. 
Similarly disposed, Coe and Fitz -Gibbon (1998) argued that even the consensus about Edmonds’ five-factor model 
(1979) and Reynold’s and Teddlie’s nine-factor model (2000) was premature, given the field’s penchant for vague 
concepts and sloppy measurement. 
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3. STUDENT CHARACTERISTICS AND PERFORMANCE 

IN 2005-06 



This chapter describes students who were enrolled in and attending the 72 diploma- 
granting NCHS high schools as of October 31, 2005.^ NCHS students are described relative to 
the full population of New York City public high school students active on October 31, 2004, the 
most recent date for which system-wide data were available at the time of this evaluation’s data 
analysis. 



Characteristics of NCHS and New York City Students 
in 2005-06 

Student profiles are presented in Exhibit 2. The discussion of these data is informal and 
relies on inspection rather than statistical analyses because NCHS students are included in the 
city wide counts. 

Forty-two percent of NCHS students were in the ninth grade in 2005-06. This was 
disproportionately high relative to citywide patterns (33 percent) but expected, given that most 
NCHS schools were in start-up mode, adding a grade per year. 

NCHS schools enrolled a higher proportion of females than the city (53 percent versus 49 
percent) and higher proportions of African American and Hispanic students (93 percent versus 
73 percent). Substantially higher proportions of NCHS students were eligible for free and 
reduced-price meals (82 percent versus 47 percent), although DOE officials indicated that both 
eligibility rates were likely to be under-estimates because students in this age group often fail to 
request assistance. 

Compared to city wide rates, slightly lower proportions of NCHS students were English 
Eanguage Eearners (10 percent versus 12 percent) and new immigrants (6 percent versus 8 
percent). The proportions of students receiving special education services were very similar in 
NCHS schools and city wide (10 percent versus 1 1 percent). 

Middle- school achievement was lower among students who entered NCHS schools. In 
the eighth grade, about 24 percent of NCHS students were proficient or advanced in English 
language arts (EEA), compared to 36 percent city wide. About 30 percent of NCHS students 
were proficient or advanced in math, compared to 42 percent citywide. 



^ As of 2005-06, three of the 75 NCHS secondary schools served students through eighth grade only. We note as 
well that three NCHS schools opened in September 2005; those schools are not included in this evaluation. 



8 




Exhibit 2 

Characteristics of NCHS and New York City High Schooi Students, in Percents 



Characteristics 


NCHS 2005-06 


NYC 2004-05 

(N = 307,899)^ 


All Grades 


(N= 18,466) 


(N = 276,279f 


Grade 9 


41.5 


32.6 


Grade 10 


36.4 


28.1 


Grade 1 1 


15.3 


17.2*’ 


Grade 12 


5.3 


15.3*’ 


Ungraded 


1.5 


6.8'’ 


Gender 


(N= 17,885) 




Female 


52.5 


49.1 


Male 


47.5 


50.9 


Race 


(N= 17,691) 




African American 


43.0 


35.0 


Asian and Others 


4.4 


14.4 


Hispanic 


49.6 


37.5 


White 


3.0 


13.3 


Eligible for ELL 


(N = 18,466) 




Yes 


9.9 


12.3‘’ 


Eligible for Free/Reduced Price Lunch 


(N = 18,466) 




Eligible 


82.2 


46.8 


Ineligible 


9.1 




Missing Data 


8.7 




Recent Immigrant 


(N = 18,466) 




Yes 


5.8 


7.8® 


Special Education Flag (Integrated and 
Other) 

Yes 


(N = 18,466) 
9.8 


10.5 


Eighth-Grade ELA Proficiency 

Proficient/Advanced 


(N =14,339) 
24.0 


35.5' 


Eighth-Grade Math Proficiency 

Proficient/Advanced 


(N= 15,103) 
29.7 


42.0' 



® Data for total enrollment were obtained from the DOE official audited October 31 , 2006, register 
http://sdat.nycboe.net . These data are the basis for all demographic breakouts unless otherwise indicated. 

‘’Data for total enrollment disaggregated by grade employed a lower base (276,279); these data too were obtained 
from the DOE official audited October 31 , 2006, register http://sdat.nvcboe.net . 

‘’Citywide ELL data were obtained from Appendix 4 of ELLs in New York City: Student Demographic Data Report 
(New York City Department of Education, Summer 2006). 

“ Citywide data were obtained from DOE 2004-05 School Report Cards. These data do not appear to be reliable. 
The proportion of New York City students in the eighth grade in 2005-06 receiving free- or reduced-price lunch was 
84 percent, according to the 2005 Trial Urban District Assessment {December 2005). Any under-reporting would 
presumably apply to all students, including NCHS students. 

® Citywide recent immigrant data were obtained from Chart 30 of ELLs in New York City: Student Demographic Data 
Report (New York City Department of Education, Summer 2006). 

‘ Citywide proficiency rates are reported for entering ninth- and tenth-graders, as presented in DOE 2004-05 School 
Report Cards. 
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Performance of NCHS Students in 2005-06 



We examined the school attendance, promotion rates, and suspension rates of NCHS 
students in 2005-06. The average daily attendance rate of NCHS students (including students 
in un-graded special education classes) was 84 percent; the median was 91 percent. Depending 
on which marker you choose, the NCHS initiative is either substantially below or very much on 
target with regard to its 92 percent daily attendance goal. The mean 2005-06 NCHS daily 
attendance rate compared favorably to the attendance rate of high school students city wide, 
which was 81 percent. The 2005-06 NCHS rate was, however, a drop from the prior year, when 
the average was 86 percent. Attendance rates in 2005-06 were higher in NCHS schools relative 
to city averages at ninth and tenth grade and in un-graded special education, almost identical in 
eleventh grade, and lower in twelfth grade. ^ 



Exhibit 3 

NCHS and New York City High Schooi Students, 
Average Daiiy Attendance Rates, by Grade 



Grade 


NCHS 2005-06 

(N =18,466) 


NCHS 2005-06 

(N =18,466) 


NYC 2005-06 

(N =283,391) 




Mean ADA 


Median ADA 


Mean ADA 


All 


83.9% 


91.0% 


81.2% 


Grade 9 


82.7% (n=7,659) 


90.8% (n=7,659) 


79.0% 


Grade 10 


84.6% (n=6,719) 


91.0% (n=6,719) 


83.3% 


Grade 1 1 


86.3% (n=2,829) 


91 .8% (n=2,829) 


86.4% 


Grade 12 


84.7% (n=979) 


89.0% (n=979) 


86.3% 


Un-graded Special Education 


69.7% (n=280) 


81 .2% (n=280) 


67.6% 



Of the 6,940 ninth-grade students who enrolled in NCHS schools in 2005-06 and 
remained active in the New York City school system in general education in 2005-06, 80 percent 
were promoted, as indicated by their 2006-07 grade status, a drop from the prior year’s 84 
percent promotion rate. The city wide ninth-grade promotion rate in 2004-05 was 72 percent. 

The rates for NCHS tenth- and eleventh-graders were, respectively, 76 percent (of 6,145 
students) down from 80 percent and 89 percent (of 2,617 students) down from 93 percent. 

NCHS students had high suspension rates. Of the 18,466 students enrolled in 2005-06, 
1,440 or 7.8 percent were suspended. This is higher than the 2005-06 citywide rate of 6.5 
percent, which we calculated based on data provided by DOE. Among general education 



^ Eleventh- and twelfth-grade attendance patterns of NCHS students may reflect the presence of students who 
would not be in school at all if they were enrolled in New York City high schools with less holding power. 



10 








students in NCHS schools, suspension rates were higher for ninth-grade (9 percent) and tenth- 
grade students (8 percent), and lower for eleventh-grade (6 percent) and twelfth-grade students (3 
percent). About 8 percent of NCHS students in un-graded special education were suspended. 
Suspension rates were lower in NCHS schools before 2005-06. Of the 1,169 students enrolled in 
2002-03, 2 percent were suspended that year. Of the 4,258 students enrolled in 2003-04, 4 
percent were suspended in that year. Of the 12,1 1 1 students enrolled in 2004-05, 6 percent were 
suspended."^ 



In this instance as well, NCHS suspension rates may be high relative to the city because suspension-prone students 
may be remaining active in their NCHS schools (i.e., they are enrolled and attending school), rather than dropping 
out. Also, NCHS schools may be setting and enforcing higher behavioral standards than other high schools. 
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4. EDUCATIONAL ENVIRONMENT IN NCHS SCHOOLS 



This chapter describes the methods we used to assess the eomponents of the schools’ 
educational environments that were targeted in the initiative’s change theory. It also describes 
findings regarding ehange over time in eaeh component and in our composite measure, whieh 
eharaeterizes the quality of sehool-level instructional systems. 



Methods 

Sample 

Annually from 2002-03 through 2005-06, we administered surveys to students, teachers, 
prineipals, and community-partner representatives in NCHS schools serving high school 
students. There were 72 such NCHS schools in the evaluation sample in 2005-06. In that year, 
staff members in three schools failed to cooperate with survey administration protocols. The 
proportions of school staff and partners providing data were as follows: principals, 81 percent; 
community partners, 76 pereent; teachers, 73 percent; and students, 73 percent. 



Instruments 

Survey instruments were constructed to elieit respondents’ pereeptions of their sehools’ 
functioning relative to the organizational prineiples that formed the NCHS aetion theory (Exhibit 
1). In some cases, single items represented a line of inquiry. In most cases, constructs were 
examined using multi-item indexes. 



Analysis 

In analyzing survey data, we worked iteratively on several fronts. We grouped items 
based on their apparent alignment with eaeh of the 10 NCHS program principles and then again 
based on output from factor analysis exercises. We pruned groups to exclude items with poor 
response variation. We examined shortened lists for internal consistency (Cronbach’s alpha), 
and we trimmed those lists to items that optimized eohesion. 

All items ultimately included in each index carried the same weight. When a respondent 
affirmed that a condition aligned with an NCHS founding principle existed in his or her school, 
we awarded one point. We added points aeross items and raters, and divided sums by the 
number of items multiplied by the number of raters. This proeess placed all indexes on a seale 
that ranged from 0.00 to 1.00, with higher scores representing more favorable ratings. A score of 
.80 means that 80 percent of responses were affirmative; a score of .40 means that 40 percent of 
responses were affirmative. 
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At the conclusion of these analyses, we grouped seven indexes into a composite measure, 
labeled the “quality of instructional systems index,” based on their high degree of conceptual and 
statistical relatedness. This composite includes indexes of Regents alignment, educational focus, 
principal leadership of instruction, quality of professional development, quantity of professional 
development, teacher influence, and teacher collaboration on instruction. 

In analyses, we arrayed responses to questions and indexes in two ways. First, we 
clustered data for all schools by school year, irrespective of each school’s initial year of 
operation, to determine the overall environmental patterns of NCHS schools. Second, we 
clustered data for each of three groups of schools, with the groups determined by the school year 
in which the school began operation. 



Data Display 

Tables included in this chapter provide the following information: (1) a list of items 
included in questions and indexes, (2) the respondent group who provided data (principals, 
teachers, etc.), and (3) for indexes, reliabilities in the form of Cronbach’s alpha. Like other 
reliability coefficients, Cronbach’s alpha ranges from 0.00 to 1.00. Scores toward the high end 
of the range (above 0.70) suggest that items in the index are measuring the same thing. 

Following the tables that list survey items, two figures are presented. The first figure 
depicts respondent ratings of the construct over time /or all NCHS schools by school year. The 
second figure depicts ratings over time within groups of NCHS schools. Circles, triangles, and 
squares represent index means, and vertical lines above and below represent the estimated 95 
percent confidence intervals of means. When confidence bands for two means do not overlap, 
the means are statistically different. Appendix A supplements this chapter by presenting 
descriptive statistics for each data element depicted in the exhibits that follow. 



Overview of Findings 

We focused analyses on two questions: 

■ Did stakeholders’ ratings of school conditions improve over time across schools? 

■ Did stakeholders’ ratings improve over time within school groupings?^ 

The second question was a nuanced version of the first question in that it controlled for 
the number of years NCHS schools had been open. A brief summary of findings is presented 
immediately below, followed by a detailed presentation of each index. 



^ We defined school groups by inaugural years. Group 1 schools were those that opened in 2002-03, Group 2 
schools were those that opened in 2003-04, and Group 3 schools were those that opened in 2004-05. 
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Did Ratings Improve Across Schoois Over Time? 

Whether initial ratings of school conditions were low and suggested vulnerabilities (as 
they did in areas such as school facilities, quality and quantity of professional development, and 
parental involvement) or they were high and suggested strengths (as they did in areas such as 
teacher expectations, educational focus, and teacher- student relations), ratings were largely 
constant over time. 

In one domain, however, ratings clearly improved. Teachers provided successively 
higher ratings of the alignment of instruction with Regents requirements over time. 

On two indexes and one item ratings clearly fell over time. Teachers provided 
successively lower ratings of student discipline and their own influence on school policy and 
curriculum. And they indicated that they were teaching increasing numbers of students over 
time. Ratings of principal leadership and teacher collaboration drifted downward after 2003-04, 
but those changes were not statistically significant. 



Did Ratings Improve within Groups of NCHS Schoois over Time? 

The change in Regents alignment noted earlier was more of a between-group than a 
within-group phenomenon. Although some increase in Regents focus occurred within school 
groupings, most of the change occurred across successive groups of schools. Schools that 
opened in later years had higher starting points on this school feature and maintained that 
advantage. 

On the other hand, the trends earlier noted in student discipline and teacher influence 
occurred irrespective of school groups. Within each set of schools, student discipline and teacher 
influence declined, suggesting the possible influence of increasing school size. (These matters 
are taken up in subsequent chapters.) Parental involvement and teacher leadership of advisory 
groups also seemed to decline in Group 2 and Group 3 schools over time, but those patterns were 
not observed in Group 1 schools. 



Findings 

1. Rigor 

This construct was measured by one item and one scale. The item measures students’ 
perceptions of teacher expectations. The scale examines teachers’ assessments of the alignment 
of school curricula with Regents standards. 

Teacher expectations. The question that addressed students’ perceptions of teacher 
expectations is presented immediately below. 
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Item 1 


Survey 


Teacher Expectations 


Student 



Do you agree with the following statement about academic expectations at this school? 



My teachers have high standards for my academic performance 



There was eonsistently high agreement with this statement over the years of the initiative. 

Teacher Expectations (all schools) Teacher Expectations (by school group) 





Regents alignment. We asked teaehers about their individual efforts to align instruetion 
with Regents requirements for high sehool graduation. Responses to items ineluded in the index 
were closely related, as indicated by a high Cronbach’s alpha. 



Index 1 


Survey 


Alpha 


Regents Alignment 


Teaeher 


0.89 


Have the Regents standards been used in the following tasks. . . 




Selecting curricular materials 

Developing curriculum 

Designing classroom assessments 

Developing a school Improvement plan 

Designing or selecting professional development opportunities 





The graphic to the left on the following page shows gradual increases over time in the 
alignment of instruction in NCHS schools with Regents requirements, resulting in a significant 
overall increase in alignment between 2002-03 and 2005-06. The graphic to the right shows that 
instruction in Group 2 schools was from the outset better aligned with Regents requirements than 
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was instruction in Group 1 schools. By 2005-06 alignment was statistically higher in both Group 
2 and Group 3 schools than in Group 1 schools, although alignment also appeared to increase in 
Group 1 schools. 

Regents Alignment (all schools) Regents Alignment (by school group) 





2. Personalized Relationships 

This construct was measured by two scales and two variables. The first scale measured 
teachers’ perceptions of student discipline. The second scale measured students’ perceptions of 
teacher-student relationships. The first variable reported on the percentage of teachers who 
facilitated advisory groups, and the second variable identified the mean numbers of individuals 
whom instructors reported teaching each week. 

Student self -management. The seven items framing teachers’ perceptions of student 
behavior are presented below. Responses to items included in the index were closely related, as 
indicated by a strong Cronbach’s alpha. 



Index 2 


Survey 


Alpha 


Student Self-Management 


Teacher 


0.82 


Are the following positive conditions present in your school? 






Low level of student absenteeism 

Low level of students cutting class 

Low level of physical conflicts among students 

Low level of verbal abuse of teachers by students 

Student respect for teachers 

Student respect for other students 

Low level of student apathy 
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The graphic to the left shows that teachers perceived a sharp and a statistically significant 
decline in student discipline after 2003-04. The graphic to the right shows that the perceived 
decline occurred within school groups. This suggests that school-level factors, such as, for 
example, increasing school enrollments, were more likely drivers than were exogenous events 
that occurred in specific years. 



Student Self-Management (all schools) Student Self-Management (by school group) 





Teacher-student relationships. The six items asking about students’ perceptions of 
teacher expectations are presented below. Responses to items included in the index were closely 
related, as indicated by the relatively high Cronbach’s alpha. 



Index 3 


Survey 


Alpha 


Teacher Student Relationships 


Student 


0.81 


How much do you agree with the following statements about teachers in this school? 






Teachers in this school treat me with respect 

1 feel that 1 can talk to the teachers in this school about things that are bothering me 

Teachers in this school really care about me 

Teachers in this school value my opinions 

Teachers in this school always try to be fair 

1 feel safe and comfortable with the teachers in this school 





Students displayed consistently high agreement with this statement over the years of the 
initiative. Ratings of students in Group 2 schools were, however, significantly lower than ratings 
of students in Group 1 and Group 3. 
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Teacher Student Relations (all schools) 



Teacher Student Relations (by school group) 





Advisories. We asked teachers whether they led student advisory periods. This single 
item is presented below. 



Item 2 


Survey 


Advisory Groups 


Teacher 


Do you currently lead an advisory period? 



The graphic to the left shows decreasing rates of teacher leadership of advisory periods 
between 2002-03 and 2005-06. The graphic to the right shows that the decline was not due to 
lower starting points in successive school groups nor was it universal. The decline occurred only 
in Group 2 and Group 3 schools, possibly because adults other than teachers (e.g., counselors, 
staff of partner organizations) led at least some advisories in those schools. 



Advisory Groups (all schools) 




Advisory Groups (by school group) 
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Students taught. A question asked teachers to identify the numbers of students (unique 
individuals) whom they teach each week, on average. 



Item 3 


Survey 


Number of Students Taught 


Teacher 


How many different students do you teach each week? 




(enter number of students) 





The numbers of students whom teachers taught appeared to increase gradually. Student 
loads in 2005-06 were statistically higher than in 2002-03. The change resulted from gradual 
increases within and between groups. 



Students Taught (all schools) 




Students Taught (by school group) 




3. Clear Focus 

This construct was measured by a single scale. The scale assessed teachers’ perceptions 
of the coherence of educational intentions in the school. The five items in this scale are 
presented below. Responses to questions included in the index were loosely related, as indicated 
by a relatively low Cronbach’s alpha. 
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Index 4 


Survey 


Alpha 


Educational Focus 


Teacher 


0.60 


Do you agree with the following statements about the educational focus of your school? 






1 understand and support this school’s educational focus 
The school’s educational focus is closely coordinated across grades 
There is a common set of classroom assessments that 1, and all teachers, use 
There is a core curriculum that 1, and all teachers, follow 

My course content and instructional materials reflect the school’s educational focus 





Teacher ratings of instructional focus were high and largely consistent across time and 
school groups. 

Educational Focus (all schools) Educational Focus (by school group) 





4. Instructional Leadership 

This construct was measured by a single scale. The scale assessed teachers’ perceptions 
of principal leadership of the instructional program. Responses to the five items included in the 
index were closely related, as indicated by a relatively high Cronbach’s alpha. 
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Index 5 


Survey 


Alpha 


Principal Leadership 


Teacher 


0.83 



Do you agree with the following statements about leadership in this school? 



My principal monitors the curriculum I use In my classroom to see that It reflects my school’s educational 
focus 

My principal monitors my classroom Instructional practices to see that they reflect the school’s educational 
focus 

My principal evaluates my performance using criteria directly related to the school’s educational focus 
My principal Is available to provide me with guidance and assistance In structuring my instructional 
practices to reflect the school’s educational focus 

My principal Informs teachers about our progress In meeting our school goals 



Teachers’ ratings of principal leadership of instruction were high and roughly consistent 
across time and school groups. 



Principal Leadership (all schools) 




Principal Leadership (by school group) 




5. Professional Development and Collaboration 

This construct was measured by four scales, two relating to professional development and 
two relating to professional collaboration. 

Quality of professional development. This construct was measured by a single scale, 
which assessed teachers’ perceptions of the alignment of the professional-development routines 
in their school with best practices. Responses to the five items included in the index were 
adequately related, as indicated by an acceptable Cronbach’s alpha. 
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Index 6 


Survey 


Alpha 


Professional-Development Quality 


Teacher 


0.77 


The professional development 1 received has usually or always. . . 

Included feedback and guidance while 1 was trying new strategies in my classroom 


Reflected my input into the design and content of the activities 
Included opportunities to work productively with colleagues in my school 
Helped my school staff work together better 

Do you have some or major influence at this school in the following areas? 

Determining the content and design of professional development for teachers 





Teachers’ ratings of the quality of professional development activities were neither high 
nor improving across time and school groups. 

Professional-Development Quality (all schools) Professional-Development Quality (by school group) 





Amount of professional development. This construct was measured by a single scale. 
The scale asked teachers to identify areas of professional practice in which they had received 
more than four hours of structured assistance. The focus was on areas of practice that were 
expected to have general relevance in NCHS schools: subject content, assessment, thematic and 
interdisciplinary teaching, teaching low achievers, and the promotion of literacy across content 
areas. Responses to the five items included in the index were related, as indicated by an 
acceptable Cronbach’s alpha. 
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Index 7 


Survey 


Alpha 


Professional-Development Quantity 


Teacher 


0.75 


Have you received four or more hours of professional development in the following areas this academic year? 


Subject-specific content or instructional strategies 
Strategies for teaching low achieving students 
Methods for interpreting and using assessment data 
Methods for teaching literacy across content areas 
Methods for thematic and interdisciplinary teaching 





Positive responses regarding the availability of staff development occurred in about 40 
percent of cases. These relatively low ratings regarding structured staff development 
opportunities were consistent across years and school groups. 



Professional-Development Quantity (all schools) Professional-Development Quantity (by school group) 





Teacher influence. Questions related to teachers’ perceptions of their influence on 
school policy and curriculum are presented below. Responses to the five items included in the 
index were related, as indicated by an acceptable Cronbach’s alpha. 



Index 8 




Survey 


Alpha 


Teacher Influence 




Teacher 


0.74 


Do you think you have some or major influence at this school in the following areas? 

Establishing school discipline policies 
Establishing and shaping the school curriculum 
Selecting instructional materials that support the curriculum 
Determining student retention and promotion policy 
Making staffing and/or hiring decisions 
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The graphic to the left shows consistent decreases in perceived influence over time and a 
statistically significant decrease in perceived influence between 2002-03 and 2005-06. The 
graphic to the right shows that teacher perceptions of declining influence were not associated 
with lower starting points in successive school groups. Rather the change in perceptions 
occurred over time within school groups. This pattern tends to rule out exogenous events as a 
driver and points toward factors internal to the schools, whether increasing enrollments or some 
other factor. 





Teacher collaboration. The five items tapping teachers’ experience of professional 
collaboration are presented below. Responses to questions included in the index were loosely 
related, as indicated by a low Cronbach’s alpha. 



Index 9 


Survey 


Alpha 


Teacher Collaboration 


Teacher 


0.63 



Do you agree with the following statements about the professional environment in your school? 



Most teachers share the same beliefs and values about the central mission of the school 
There is a great deal of cooperative effort among the staff members 

Do you agree with the following statements about your school? 

Teachers are continual learners and team members through professional development, common planning, 
and collaboration 

There are formal arrangements within the school that provide opportunities for teachers to discuss and 
critique their own and others’ instruction 

Would you say that the following activities were usually or always the main task of planning meetings? 

Building effectiveness as a team 



24 



Teachers’ ratings of collaboration were high and consistent over time in Group 1 and 
Group 3 schools. Ratings trended downward in Group 2 schools, although differences were not 
statistically significant. 

Teacher Collaboration (all schools) Teacher Collaboration (by school group) 





6. Meaningful Continuous Assessment 

This construct was measured by one scale examining students’ perceptions of the quality 
of teacher- made tests. 

Meaningful assessments. The three items related to students’ perceptions of the quality 
of teacher- made tests are presented below. Responses to questions included in the index were 
loosely related, as indicated by a low Cronbach’s alpha. 



Index 10 


Survey 


Alpha 


Meaningful Assessments 


Student 


0.65 


For all or most of your classes, are the following statements true about the tests your teachers give in class? 


The tests are a good measure of how much I have learned 

The tests cover the same materials that the teacher presented in class 

My teachers prepare me well for the tests 





Students’ ratings of teacher- made tests were high and consistent over time and across 
groups of schools. 
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Meaningful Assessments (all schools) 



Meaningful Assessments (by school group) 





7. Community Partners 

This construct was measured by a scale that assessed principals’ perceptions of the scope 
of community partners’ involvement in school life. The scale measuring partner involvement 
included 18 items. Responses to questions included in the index were closely related, as 
indicated by a relatively high Cronbach’s alpha. 



Index 11 


Survey 


Alpha 


Role of Community Partners 


Principal 


0.87 



Does the community partner play at least some role in the day-to-day functions of this school in these particular 
areas? 

Fund raising 

Pianning schooi budgets 

Deiivery of instruction 

Tutoring and/or mentoring students 

Academic pianning with students 

Evaiuating the overali instructionai program 

Teacher recruitment and hiring 

Student recruitment and seiection 

Professionai deveiopment pianning with teachers 

Administration 

After-schooi programming 

Curricuium design or seiection 

Determining the content and deiivery of teacher professionai deveiopment activities 

Determining specific professionai and teaching assignments 

Providing facuity professionai deveiopment 

Organization of out-of-school learning opportunities for students 

Helping out in the building (e.g., as teachers’ assistants, lunchroom monitoring) 

Communicating with parents 



26 



Responses indicated substantial variation in principals’ assessment of partner 
involvement at each measurement occasion. The apparently lower involvement of partners in 
Group 2 schools was not statistically significant. 

Community Partners’ Role (all schools) Community Partners’ Role (by school group) 





Evaluators also administered this survey scale to respondents in community partner 
organizations. Their responses were similar to those of the school principals, except that the 
community partner respondents provided consistently more positive assessments of their 
involvement in the schools than did the principals. 



8. Family/Caregiver Partnership and Involvement 

This construct was measured by one item, in which we asked teachers whether parent 
involvement was adequate in their schools. The item is presented below. 



Item 4 


Survey 


Parental Involvement 


Teacher 


Are parents sufficiently involved in your school? 



The graphic to the left shows decreases in perceived involvement after 2003-04. The 
graphic to the right shows that the decline was not due to lower starting points in successive 
school groups. Rather it was due to lesser involvement over time within Group 3 and especially 
Group 2 schools. 
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Parental Involvement (all schools) 



Parental Involvement (by school group) 





9. Youth Participation and Deveiopment 

This construct was measured by two seales. One asked students about engagement in 
sehool activities. The other asked principals about the range of non-academic learning 
opportunities in their sehools. 

Student participation in school life. The scale measuring student engagement included 
seven items. Responses to questions included in the index were related, as indicated by an 
acceptable Cronbach’s alpha. 



Index 12 


Survey 


Alpha 


Student Engagement 


Student 


0.79 



Have you done any of the following things at this school? 

Voted in student council elections 

Volunteered or been selected to work in or lead an activity (sports, club, etc.) 

Helped out in the school office (answered the phone, entered data in the computer, etc.) 

Served on a student council or leadership team for this school 

Helped plan school events and activities 

Helped with meetings for parents and community members 

Been asked by staff for feedback/comments about the school or an activity 



The graphics show that student engagement in extra-curricular activities and school life 
was not especially high or increasing, as judged by participation in the listed activities. 
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Student Engagement (all schools) 



Student Engagement (by school group) 





Extended learning opportunities . The scale measuring the availability of extended 
learning opportunities included six items and appeared in survey instruments distributed in the 
final two years of this study. Responses to questions included in the index were related, as 
indicated by an acceptable Cronbach’s alpha. 



Index 13 


Survey 


Alpha 


Learning Opportunities 


Principal 


0.71 


Are the following extended learning opportunities for students available at your school? 
Mentoring 

Service iearning or community service 
Training in youth ieadership 
Support for making informed choices 
Preparation for pubiic speaking or presentations 
Coiiege awareness activities (e.g., tours, SAT prep) 





The graphics show that the schools consistently provided extensive supplementary 
opportunities for students to learn, as assessed by school principals. Although survey items did 
not ask who provided extended learning opportunities, interview data indicated that partner 
organizations were often involved in these school operations. 
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Learning Opportunities (aii schoois) 



Learning Opportunities (by schooi group) 





10. Effective Uses of Technology and Other Resources 

This construct was assessed by one scale measuring the use of technology and six items 
measuring the availability of technology and other school resources. 

Use of technology . The scale measuring the use of technology included only two items. 
Responses to questions included were modestly related, as indicated by a low Cronbach’s alpha. 



Index 14 


Survey 


Alpha 


Computer Use 


Teacher 


0.66 


Do you agree with the following statements about instruction using computers? 




In my classes, I teach students how to use computers through their course work 
In my classes I give assignments that require students to use a computer 





The graphics show moderate use of technology and no evidence of increasing use. 
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Computer Use (all schools) 



Computer Use (by school group) 





Availability of technology . The item measuring the availability of technology is 
presented below. 



Item 5 


Survey 


Technology 


Teacher 



Do you agree with the following statement about instruction using computers? 



There are enough computers available in this school for teachers to use them in instruction 



The graphics indicate that typically less than half of teachers were satisfied by the 
availability of computers. 

Technology Availability (all schools) Technology Availability (by school group) 
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Instructional materials. One item measured teachers’ assessment of the availability of 
instructional materials, and it is presented below. 



Item 6 


Survey 


Instructional Materials 


Teacher 



Do you agree with the following statement about your school? 



Necessary instructional materials are available as needed by the staff 



A high proportion of teachers (no less than 70 percent at any point) were satisfied by the 
availability of instructional resources (other than computers). 

Instructional Materials (all schools) Instructional Materials (by school group) 





Proportion of teachers with masters degrees. One item inquired about teachers’ 
academic background, and it is presented below. 



Item 7 


Survey 


Teacher Degree 


Teacher 


Have you earned a Master’s Degree or higher? 



Consistently, more than 70 percent of NCHS teachers had graduate degrees. 
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Teacher Degree (all schools) 



Teacher Degree (by school group) 





Proportion of teachers certified. One item asked about teacher certification, and it is 
presented below. 



Item 8 


Survey 


Teacher Certification 


Teacher 


Do you hold the following types of teaching certificates or licenses? 


New York State Permanent Certificate 





NCHS teachers were new to their profession, and, as indicated below, consistently less 
than 30 percent were fully certified teachers. 

Teaching Certificate/License (all schools) Teaching Certificate/License (by school group) 
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Facilities. One item asked about the adequacy of school facilities, and it is presented 

below. 



Item 9 


Survey 


Facilities 


Teacher 


Is there adequate physical space at your school? 



Consistently less than 50 percent of teachers felt that building conditions were adequate. 

Adequacy of Facilities (all schools) Adequacy of Facilities (by school group) 





Host school. One question asked about relationships with host schools. 



Item 10 


Survey 


Host School 


Teacher 


Are relationships with the host school working out? 



Nearly 60 percent of teachers were satisfied with host-school relationships. 
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Host School (all schools) 



Host School (by school group) 





11. Quality of Instructional Systems 

Seven of the foregoing indexes drawn from the teacher survey, because of their high 
degree of statistical relatedness, were ultimately grouped into a composite measure, described 
here as an “index of the quality of instructional systems.” This composite measure includes 
indexes of Regents alignment, the coherence of the school’s educational focus, principal 
leadership of the instructional program, the quality of professional development, the quantity of 
professional development, teacher influence, and teacher collaboration on building the 
instructional program. Two other teacher scales (Student Discipline and Use of Technology), 
although correlated with the composite index, were not included in the scale to maintain the 
measure’s conceptual integrity. 



Index 15 (Composite) 


Survey 


Alpha 


Quality of Instructional Systems 


Teacher 


0.88 


Components 


Index Number 


Alpha 


Regents Alignment 


1 


0.89 


Educational Focus 


4 


0.60 


Principal Leadership 


5 


0.83 


Quality of Professional Development 


6 


0.77 


Quantity of Professional Development 


7 


0.75 


Teacher Influence 


8 


0.74 


Teacher Collaboration on Instruction 


9 


0.63 



The graphics show a steady pattern of quality in instructional systems over time and 
school groups. 
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Quality of Instructional Systems (all schools) 



Quality of Instructional Systems (by school group) 
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5. CLASS OF 2006 GRADUATION RATES 



In this chapter and the next, we examine achievement in NCHS schools from multiple 
perspectives. This chapter compares the graduation rate of students in the NCHS Class of 2006 
with the New York City Class of 2005 overall and with students in the Class of 2006 in matched 
comparison-group schools. In Chapter 6, as a way of anticipating future graduation rates, we 
compare the achievement of sampled students in the NCHS Class of 2006 with sampled students 
in upcoming classes (Class of 2007 through Class of 2009). 

We found that graduation rates for the NCHS Class of 2006 exceeded comparisons by 18 
to 20 percentage points. The four-year graduation rate of students who transferred from NCHS 
schools to other city high schools was much lower than was the graduation rate of students who 
stayed in NCHS schools. However, relatively few students who enrolled in NCHS schools as 
ninth-graders later transferred to other high schools. 

As described in the next chapter, future NCHS classes differed from the Class of 2006 in 
their prospects for educational success, based on their students’ average achievement prior to 
high school and their current achievement in NCHS schools. 



Methods 

Every year DOE computes a graduation rate for students in general education classes, 
which includes students who receive special education services in “a less restrictive setting.” 

It also calculates a graduation rate for students with severe disabling conditions, the Special 
Education Class. Here we report on the graduation rate of students in general education classes, 
both city wide (inclusive of NCHS schools), in NCHS schools, and in comparison-group schools. 

We calculated graduation rates for NCHS schools following DOE standards and 
procedures in place through 2005. At that time, DOE calculated graduation rates for general 
education students in two steps. Eirst, it defined a “class” based on students’ projected four-year 
graduation dates. Most students assigned to the Class of 2006, for example, entered the class as 
ninth-graders in 2002-03. The remainder transferred into the DOE system later in high school: 
as tenth-graders in 2003-04, eleventh-graders in 2004-05, or twelfth-graders in 2005-06. 

Having thus defined a class, the school system next determined the status of students at 
their expected graduation date (defined not as June but as August of the graduation year to allow 
for summer-school graduates), using four categories: graduated, still enrolled, dropped out, and 
discharged. Graduates were students who received a high school diploma, GED, or special 
education certificate. Still-enrolled were students who remained on the school system’s register. 
Dropouts were students who left the system without enrolling in another school system or 
diploma program. Discharges were students who left the system to enroll in another education 
program or setting, who reached the age of 21, or who died prior to completing high school. 
DOE calculated graduation rates exclusive of discharges, such that the denominator for the 
graduation -rate calculation was the sum of graduates, students still enrolled, and dropouts. 
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In The Class of 2006 Four-Year Longitudinal Report, published in May 2007 (subsequent 
to our data analysis), the New York City DOE revised its longstanding policy of counting 
students who received GEDs through its auspices and students who completed summer school 
successfully as graduates. Because our 2006 NCHS graduation rates were calculated according 
to doe’s earlier protocols, we contrast NCHS graduation rates for 2006 with citywide patterns 
in 2005. 

Ten of the twelve NCHS schools that opened in 2002-03 are included in this analysis.^ 
Excluded schools (Community Prep and South Brooklyn) did not enroll a full class of ninth- 
graders in 2002-03. 



Findings 

Citywide Graduation Rates 

More than one-half (58 percent) of New York City students in the Class of 2005 
graduated in four years, as shown in Exhibit 4. This was the highest on-time graduation rate 
since DOE began reporting these statistics over 20 years ago. Twenty-seven percent of the Class 
of 2005 remained enrolled in high school for a fifth year. Eess than 15 percent of the Class of 
2005 had dropped out by the end of four years. 

Exhibit 4 

Ciass of 2005 Four-Year Outcomes, Citywide, in Percents 



Status 


Class of 2005 

(N= 65,705) 


Graduates of 2005 

(N = 38,240) 


Graduates 


58.2 




Regents Diploma 




35.9 


Advanced Regents Diploma 




24.8 


Local Diploma 




34.8 


G.E.D. 




4.5 


Still Enrolled 


27.0 




Dropouts 


14.8 





As shown in Exhibit 4, over a third of graduates (36 percent) received a Regents diploma. 
Beginning with the Class of 2005, a Regents diploma was achieved by earning appropriate 
credits and passing five Regents exams (English, Math, Global History & Geography, a science 
Regents exam, and U.S. History & Government) with a score of 65 or above. Previously, a 
Regents diploma required eight Regents with a score of 65 or above. The next most frequent 



® Those schools are as follows: Millennium, Bronx High School of Visual Arts, Marble Hill, Mott Haven, School 
for Excellence, Bronx Leadership Academy, Bronx Guild, Bronx International High School, High School for 
Teaching and the Professions, and Community High School for Social Justice. 
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graduation status among 2005 graduate city wide was a local diploma, earned by 35 percent of all 
graduates. A local diploma was achieved by earning appropriate credits and obtaining a score of 
55-64 on five Regents exams. Almost one-quarter of graduates (25 percent) received an 
Advanced Regents diploma. Beginning with the Class of 2005, an Advanced Regents Diploma 
was achieved by earning appropriate credits and obtaining a score of 65 or above on eight 
Regents exams. A relatively small proportion of students (4.5 percent) graduated with an 
equivalency diploma. 



NCHS Graduation Rates 

As shown in Exhibit 5, 78 percent of students in the NCHS Class of 2006 graduated in 
four years. Almost 19 percent of the Class of 2006 remained enrolled in high school for a fifth 
year. Three percent of the Class of 2006 had dropped out within four years. 

The four-year graduation rate for the NCHS Class of 2006 was 20 percentage points 
higher than the city wide graduation rate for the Class of 2005. This difference in graduation 
rates derives from both lower dropout rates in NCHS schools (3 percent versus 15 percent) and 
faster completion rates (19 percent “still enrolled” in NCHS schools versus 27 percent citywide). 

A higher percentage of graduates of NCHS schools received local diplomas (52 percent 
of total graduates) than received those less challenging diplomas citywide (35 percent). And as 
would follow, a lower percentage of NCHS graduates received Regents or Advanced Regents 
diplomas (46 percent) than received those diplomas city wide (61 percent of graduates), as shown 
in Exhibits 4 and 5. When the unit of comparison is students rather than graduates, however, a 
slightly higher percentage of NCHS students in the Class of 2006 received a Regents or 
Advanced Regents diploma, compared to New York City students in the Class of 2005 (36 
percent versus 35 percent). 



Exhibit 5 

Ciass of 2006 Four-Year Outcomes, NCHS Schoois, in Percents 



Status 


Class of 2006 

(N = 728) 


Graduates of Class of 2006 

(N =569) 


Graduates 


78.2 




Regents Diploma 




43.6 


Advanced Regents Diploma 




2.6 


Local Diploma 




52.4 


G.E.D. 




0.0 


I.E.P. Diploma 




1.2^ 


Special Education Diploma 




0.2® 


Still Enrolled 


18.8 




Dropouts 


3.0 





® Eight students who entered the Class of 2006 transferred to the Special Education Class of 2006. 
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Graduation Rates in Comparison-Group Schoois 

The preceding comparisons are relevant to an assessment of the NCHS schools, but they 
are not ideal from a methodological perspective. The “classes” discussed were a year apart, and 
the students compared were not matched based on their demographic characteristics and their 
prior achievement. We conducted a second set of analyses that controlled for students’ prior 
characteristics and found persuasive although slightly smaller differences in the performance of 
NCHS students and comparison-group students. Graduation rates among students in NCHS 
schools were 18 percentage points higher than graduation rates among students in matched 
schools, as shown in Exhibit 6. 



Exhibit 6 

Ciass of 2006 Four-Year Outcomes, Comparison Schoois, in Percents 



Status 


Class of 2006 

(N = 4,228) 


Graduates of Class of 2006 

(N = 2562) 


Graduates 


60.6 




Regents Diplema 




47.4 


Advanced Regents Diploma 




19.6 


Local Diploma 




30.7 


G.E.D. 




0.1 


I.E.P. Diploma 




0.5 


Other Diploma 




1.7 


Still Enrolled 


22.1 




Dropouts 


17.3 





Our matching procedures were as follows. First, we limited possible matches to New 
York City high schools that enrolled over 750 students (as compared to the 400 students enrolled 
in NCHS schools). Next, we used optimal matching procedures (Bergstralh, Kosanke, & 
Jacobsen, 1996) to identify 10 larger schools most like the NCHS schools in our sample. The 
variables we used for matching were Class of 2006 eighth-grade math and reading scores on 
standardized tests, percent Black/Hispanic, percent new immigrant, percent English Eanguage 
Eearners, percent free- or reduced-price lunch, percent female, average age, and percent eligible 
for special education. Subsequent to the identification of optimal matches,^ we conducted 
analyses to determine if there were any statistically significant differences between the two 
groups of schools. We found no statistically significant differences (p = 0.05) at the school level. 
There were, however, differences at the student level. Comparison-group students had 
somewhat better prior achievement in math, a lower proportion of comparison-group students 
were African-American, and a lower proportion of comparison-group students were recent 



^ The comparison-group schools identified through this procedure were Cobble Hill School of American Studies, 
Health Professions and Human Services, Manhattan Center for Math and Science, High School for the Humanities, 
Washington Irving High School, Louis D. Brandeis High School, A. Philip Randolph High School, Long Island City 
High School, Flushing High School, and Adlai E. Stevenson High School. 
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immigrants. A possible implication of these findings is that comparison-group students were 
somewhat more advantaged than NCHS students (see Appendix B). 

As shown in Exhibit 6, 60.6 percent of students in the Class of 2006 in comparison-group 
schools graduated in four years, compared to 78.2 percent in NCHS schools. About 22 percent 
of students in comparison-group schools remained enrolled in high school for a fifth year, as 
compared to 19 percent in NCHS schools. In comparison-group schools, 17 percent of students 
in the Class of 2006 dropped out, compared to 3 percent in NCHS schools. 

As we saw in comparing NCHS graduates with graduates city wide, a higher percentage 
of graduates of NCHS schools received local diplomas (52 percent) than received those less 
challenging diplomas in comparison-group schools (31 percent). And as would follow, a lower 
percentage of NCHS graduates received Regents or Advanced Regents Diplomas (46 percent) 
than received those diplomas in comparison-group schools (67 percent), as shown in Exhibits 5 
and 6. When the unit of comparison was students rather than graduates, however, the 
comparison was less stark. Thirty-six percent of NCHS students in the Class of 2006 received a 
Regents or Advanced Regents diploma, compared to 41 percent of students in comparison-group 
schools. 



Graduation Rates of NCHS Transfer Students 

Noting statistically significant and important differences in graduation-related outcomes 
between NCHS and other schools, we were curious about variations in transfer rates, because the 
latter metric is not a factor in calculating “class” outcomes. We then examined outcomes for 
transfers and non-transfers from NCHS schools and the characteristics of both groups. 

Exhibit 7 presents outcomes (as of October 31, 2006) for students who entered the Class 
of 2006 in NCHS schools in 2002-03 (N = 855). Over the next four years, 179 of these students 

o 

(25 percent) transferred to other city high schools. The transfer rate in comparison-group 
schools was 31 percent, indicating greater mobility in comparison-group schools. 

The four-year graduation rate for Class of 2006 persisters in NCHS schools was 84 
percent [475/(675-1 1 1)]. The four-year graduation rate for ninth-grade entrants who later 
transferred was 35 percent [58/(179-12)]. These findings are somewhat surprising, given that 
other research has tended to find beneficial effects of school mobility (Swanson & Schneider, 
1999). 



Exhibit 8 presents the demographic and educational characteristics of students who 
remained in the NCHS schools in which they started ninth grade and the demographic 
characteristics of students who eventually transferred to other schools. The surprising finding, 
given the group’s dramatically different outcomes, was their many similarities. 



Thirty percent of NCHS transfers were to alternative schools, 42 percent went to other NCHS schools, and 28 
percent went to other city high schools. 
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Exhibit 7 

2002-03 Ninth-Grade Entrants to NCHS Schoois, Four-Year Graduation Status' 



Total new ninth-graders enrolled in NCHS schools as of October 31 , 2002 


855 


Number not changing schools before October 31 , 2006 


675 


Number changing schools before October 31 , 2006 


179 


Number missing data for School Assignment/Graduation Status as of October 31 , 2006 


1 


Status of Non-Transfers as of October 31 , 2006 


675 


Number Graduated 


475 


Number Still Enrolled 


76 


Number Dropped Out 


13 


Number Discharged Out of the School System 


111 


Status of Transfers as of October 31 , 2006 


179 


Number Graduated 


58 


Number Still Enrolled 


80 


Number Dropped Out 


29 


Number Discharged Out of the School System 


12 



® As of October 31 , 2006 



Equal proportions of boys and African American and Hispanic students were represented 
in both groups. Students in these groups entered high school at about the same age, and about 
equal proportions were proficient in math and in English Eanguage Arts. Interestingly, lower 
proportions of transfers were EEE students or in special education. 

Transfers were, however, three times more likely than non-transfers to have been 
suspended in ninth grade. And transfers had lower attendance by more than two percentage 
points the year before high school and lower attendance by almost five percentage points in ninth 
grade. 
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Exhibit 8 

Demographic Characteristics of 2002-03 Ninth-Grade Entrants to NCHS Schoois, 

by Transfer Status^ 



Characteristic 


Students Not Changing Schools 
before October 31 , 2006 


Students Changing Schools 
before October 31 , 2006 


Gender 


(N = 675) 


II 


Female 


56.0 


54.7 


Male 


44.0 


45.3 


Race 


(N = 675) 


II 


African American 


35.7 


33.0 


Aslan and Others* 


9.2 


4.5 


Hispanic 


50.8 


54.7 


White 


4.3 


7.8 


Eligible for ELL 


(N = 675) 


II 


Yes** 


13.0 


6J 


Average Age (years) 


(N = 675) 


II 


Ninth Grade 


14.4 


I 14.3 1 


Eligible for Free/Reduced Price Lunch 


(N = 675) 


II 


Eligible 


89.2 


83.2 


Ineligible 


8.3 


12.3 


Missing Data 


2.5 


4.5 


Recent Immigrant 


(N = 675) 


(N= 179) 


Yes 


13.2 


8.4 


Special Education 


(N = 675) 


II 


Yes* 


5.8 


1 2.2 1 


Suspended in Ninth Grade 


(N = 675) 


II 


Yes* 


2.1 


6.7 


Eighth-Grade ELA Proficiency 


(N = 518) 


(N=148) 


Proficient/ Advanced 


20.1 


22.3 


Eighth-Grade Math Proficiency 


{N = 532) 


(N = 149) 


Proficient/ Advanced 


20.1 


22.1 


Average Daily Attendance 


(N = 583) 


II 


Eighth Grade*** 


92.0 


1 89.7 1 


Average Daily Attendance 


(N = 675) 


II 


Ninth Grade*** 


92.4 


87.5 1 



® As of October 31 , 2006 



* p<= 0.05, ** p<=0.01 , *** p<=0.001 
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6. PERFORMANCE OF COHORTS 
FOLLOWING THE CLASS OF 2006 



This chapter presents performance data for students continuously enrolled in NCHS 
schools from ninth grade. We refer to such students relative to their expected graduation dates as 
“treatment cohorts” to distinguish them from “classes,” which also include students who transfer 
into schools after ninth grade. This chapter examines 2007, 2008, and 2009 treatment cohorts, 
relative to the 2006 treatment cohort, as a way of anticipating outcomes for those later groups of 
NCHS students. 



Findings 

Evidence suggests complex patterns of performance among later treatment cohorts. For 
example, NCHS student performance is trending downward on measures of attendance, 
suspension, and credit accumulation. Positive patterns are seen, however, in NCHS students’ 
passage of Regents tests and likelihood of obtaining Regents-endorsed diplomas. Positive and 
negative trends may be associated, at least in part, with the characteristics of the students who 
enrolled in the 2007, 2008, and 2009 treatment cohorts. The attendance and suspension records 
of these students as eighth-graders indicate increasingly high levels of education risk. Yet, in 
comparison to earlier NCHS cohorts, their eighth-grade test scores were about the same in 
reading and slightly higher in math. 

As seen in the top row of Exhibit 9, the first-year attendance rates of successive cohorts 
of NCHS students declined over the four years of NCHS operations, from 91 to 86 percent. The 
same trend is also evident among students enrolled in NCHS schools for two years and for three 
years, as seen in the second and third rows. This pattern of declining attendance raises a red flag 
about the likelihood of their on-time graduation, but facts regarding actual readiness for 
graduation are, however, more complicated. 



Exhibit 9 

Attendance Rates of Successive Cohorts 



Years Attending 


Year Entered Ninth Grade 


NCHS School 


2002-03 


2003-04 


2004-05 


2005-06 


1 


91 .4% (N=855) 


90.5% (N=2,577) 


88.8% (N=6,238) 


86.4% (N=6,323) 


2 


90.4% (N=741) 


89.2% (N=2,239) 


86.7% (N=5,304) 




3 


87.9% (N=672) 


85.3% (N=2,002) 






4 


83.0% (N=595) 









Exhibit reads: NCHS students who enroiied in ninth grade in 2002-03 attended schooi at a mean rate of 91 .4 percent 
in their first year and at a mean rate of 90.4 percent in their second year. NCHS students who enroiied in ninth grade 
in 2003-04 attended schooi at a mean rate of 90.5 percent in the first year and 89.2 percent in their second year. 
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Exhibit 10 presents data regarding suspension rates among NCHS students by cohort. 
Suspension rates of successive cohorts of NCHS students rose over the four years from 3 percent 
of ninth-grade students in 2002-03 to 7 percent of ninth-grade students in 2005-06. It is not clear 
whether these changes reflect actual increases in problem behavior in NCHS schools, a shift in 
city wide policy toward zero tolerance for serious misbehavior, or a raising of behavior standards 
in NCHS schools. 



Exhibit 10 

Suspension Rates of Successive Cohorts 



Years Attending 
NCHS School 


Year Entered Ninth Grade 


2002-03 


2003-04 


2004-05 


2005-06 


1 


3.0% (N=855) 


4.7% (N=2,577) 


6.0% (N=6,238) 


7.3% (N=6,323) 


2 


1.1% (N=741) 


6.6% (N=2,239) 


8.8% (N=5,304) 




3 


3.6% (N=672) 


7.9% (N=2,002) 






4 


2.7% (N=595) 









Exhibit reads: Three percent of NCHS students who enroiied in ninth grade in 2002-03 were suspended in that year; 
4.7 percent of NCHS students who enroiied in ninth grade in 2003-04 were suspended in that year. 



Credit accumulation trends were consistent with negative attendance and suspension 
trends, as seen in Exhibit IE Students who entered grade 9 in 2002-03 earned an average of 12 
credits in that year, whereas students who entered grade 9 in 2005-06 earned an average of 1 1 
credits in that year. 



Exhibit 11 

Credit Accumuiation by Successive Cohorts 



Years Attending 
NCHS School 


Year Entered Ninth Grade 


2002-03 


2003-04 


2004-05 


2005-06 


1 


12.0 (N=786) 


1 1 .9 (N=2,490) 


11.1 (N=5,988) 


10.9 (N=6,048) 


2 


24.1 (N=721) 


23.3 (N=2,227) 


22.1 (N=5,257) 




3 


36.0 (N=666) 


35.2 (N=2,000) 






4 


47.2 (N=595) 









Exhibit reads: Students who entered ninth grade in 2002-03 earned an average of 12 credits in that year; students 
who entered ninth grade in 2003-04 earned as average of 1 1 .9 oredits in that year. Students who entered ninth grade 
in 2004-05 earned an average of 11.1 credits in that year. Students who entered ninth grade in 2005-06 earned an 
average of 10.9 credits in that year. 



There were two important and related exceptions to these patterns of declining 
performance. Eirst, students were passing more Regents exams earlier in their high school 
careers. Exhibit 12 shows that 2003-04 NCHS entrants passed 4.5 Regents exams in three years, 
while 2002-03 entrants had passed only 4.0 Regents exams in three years. 
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Exhibit 12 

Regents Tests Passed by Successive Cohorts 



Years Attending 


Year Entered Ninth Grade 


NCHS School 


2002-03 


2003-04 


3 


4.0 (N=632) 


4.5 (N=1,941) 


4 


5.0 (N=588) 



Exhibit reads: NCHS students who enroiied in ninth grade in 
2002-03 passed 4.0 Regents after three years in high schooi. 
NCHS students who enroiied in ninth grade in 2003-04 passed 
4.5 Regents after three years in high schooi. 



Second, Exhibit 13 shows that the percentage of students prepared to graduate after only 
three years of high school with Regents diplomas increased from 28 percent of 2002-03 entrants 
to 30 percent of 2003-04 entrants. Further, the proportion of students ready for graduation after 
three years of high school increased from 38 percent of those admitted in 2002-03 to 50 percent 
of those admitted in 2004-05. A possible explanation for this pattern is that NCHS schools, after 
their start-up year, began to administer Regents exams to students earlier in their high school 
careers. Credit accumulation was not higher, but it was better focused in terms of graduation 
requirements. 

Positive and negative trends, where they existed, may be related, in part, to the changing 
characteristics of entering ninth-graders, as seen in their eighth-grade profile, which is 
summarized in Exhibit 14. Incoming ninth-graders demonstrated increasing levels of risk, as 
measured by their eighth-grade rates of school attendance and suspension. This pattern was not 
seen in students’ eighth- grade scores in reading and math, however. In fact, math scores 
appeared to be improving over time. 
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Exhibit 13 

NCHS Regents Test Performance, by Regents Dipioma Requirements 



Years Attending 
NCHS School 


Diploma Status 


Year Entered Ninth Grade 




2002-03 


2003-04 




Total number of students with Regents data 


N=632 


N=1,941 




Students meeting Regents Diploma test 
requirements 


28.2% 


29.9% 


3 


Students meeting Local Diploma test 
requirements (only) 


9.5% 


20.3% 




Students meeting either Regents or Local 
Diploma requirements 


37.7% 


50.2% 




Total number of students with Regents data 


N=588 






Students meeting Regents Diploma test 
requirements 


41 .0% 




4 


Students meeting Local Diploma test 
requirements (only) 


36.9% 






Students meeting either Regents or Local 
Diploma requirements 


77.9% 





Exhibit reads: Of students who entered ninth grade in 2002-03 and remained in an NCHS school for three years, 
28.2 percent completed requirements for a Regents Diploma. Of students who entered ninth grade in 2003-04 and 
remained in an NCHS school for three years, 29.9 percent completed requirements for a Regents Diploma. 



Exhibit 14 

Prior Performance Characteristics, by Treatment Cohort 



Eighth-Grade 


Year Entered Ninth Grade 


Characteristic 


2002-03 


2003-04 


2004-05 


2005-06 


Attendance 


91 .5% (N=739) 


91 .4% (N=2,293) 


90.4% (N=5,414) 


89.9% (N=5,537) 


Suspension 


0.5% (N=739) 


2.4% (N=2,293) 


2.3% (N=5,414) 


3.7% (N=5,537) 


Reading scale score 


685.5 (N=666) 


683.9 (N=2, 132) 


687.2 (N=5,082) 


683.7 (N=5, 134) 


Math scale score 


693.5 (N=681) 


698.7 (N=2,264) 


702.2 (N=5,332) 


701 .6 (N=5,467) 



Exhibit reads: Students who entered ninth grade in 2002-03 had an average daily attendance rate of 91 .5 percent as 
eighth-graders. Students who entered ninth grade in 2003-04 had an average daily attendance rate of 91 .4 percent 
as eighth-graders. 
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7. STATISTICAL MODELS FOR ESTIMATING 
INFLUENCES ON CREDIT ACCUMULATION 



This chapter and the next identify student and school-level factors that influenced student 
outcomes in NCHS schools. Chapter 7 presents the results of hierarchical multivariate statistical 
analyses of NCHS inputs, processes, and outcomes. Chapter 8 presents the results of similarly 
focused qualitative analyses. 



Overview of Findings 

First, as one might expect, students’ individual demographic characteristics, especially 
their age, gender, and free-lunch status, anticipated or predicted their individual achievement. 
Girls earned more credits than boys. More economically advantaged students (not eligible for 
lunch subsidies) earned more credits than lower-income students. And students who entered 
high school at a younger age (i.e., those who were less likely to have earlier been retained in 
grade) had higher credit accumulation in high school than older entrants. 

Second, and again as one might expect, academic performance prior to high school, 
especially special education status, eighth-grade math test scores, and eighth-grade school 
attendance, anticipated or predicted individuals’ high school credit accumulation. The 
differentiating effect of prior achievement was positive, meaning that students in general 
education, those with higher Grade 8 attendance, and those with higher eighth-grade math scores, 
accumulated higher numbers of credits in high school than students in special education and 
those with lower prior attendance and lower prior math scores. 

Third, schools’ group demographics — the average age of students at entry to the school, 
the percent of girls attending the school, and the percent of low-income students attending — did 
not predict individuals’ personal credit accumulation. Group-level differences in prior academic 
performance did, however, help to account for variation in individual achievement. Individual 
achievement was lower in schools that had higher concentrations of youngsters with lower prior 
math achievement and lower prior school attendance rates. 

Fourth, a single variable describing school processes was associated with variation in 
individuals’ credit accumulation. This composite variable was the average teacher rating of the 
quality of their schools’ instructional systems on a 35-item scale. Analyses indicated that 
weighted credit accumulation could be expected to increase by seven credits as a school moved 
from a score of 0.0, or lowest quality instructional systems, to a score of 1.0 or highest quality 
instructional systems. For purposes of future policy development and school support, this 
finding is particularly salient. It identifies a set of essential school processes, which are 
consistently associated with positive student outcomes and readily measurable through teacher 
surveys. 
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Fifth, several valued NCHS model components (e.g. advisories, partnerships, etc.), as 
measured in this evaluation, did not help explain variation in student outcomes across schools. 
The failure to find such relationships may, of course, be related to limitations of the measures 
used in the evaluation’s surveys and/or the modeling process. 



Methods 

Analytic Approach 

In modeling achievement outcomes, nesting matters. When individuals are gathered into 
groups such as classrooms and schools, it is appropriate to take account of the variance occurring 
at group levels as well as at the individual level. A proper statistical approach for considering 
group differences is multilevel modeling with maximum likelihood estimation (MLE). 

In practice, multi-level analyses are often performed without taking each level of nesting 
into account (Van den Noortgate, Opdenakker, & Onghena, 2005). In this research, we 
employed a two-level hierarchical model: we accounted for variation among students and 
schools but not classrooms.^ 

In preparing data for hierarchical analyses, we used SPSS (version 13.0) software. In 
hierarchical analyses, we employed hierarchical linear modeling (HLM) (version 6.1) software. 



Samples 

School level. The research sample included 64 NCHS schools. Nine of 75 NCHS 
schools did not meet the student-level data requirements for inclusion in this analysis, and four 
were missing school-level data (at least one survey data element). Given overlap in those two 
categories of missing data, we were left with a sample of 64 schools. 

Student level. Credit accumulation data were systematically available for two NCHS 
student cohorts: ninth-grade students who entered NCHS schools in 2004-05 and ninth-grade 
students who entered NCHS schools in 2005-06. We chose the former cohort for study (N = 
4,660) because credit accumulation over two years constitutes a more reliable indicator of 
success than credit accumulation over one year. All students with the demographic and 
achievement data necessary for analyses (83 percent of the population) were included in the 
study (N = 3,864).“ 



^ The classroom level is a particularly demanding one at which to work because students in high schools are cross- 
nested (i.e. they are enrolled in multiple classrooms). 

Three were middle schools, one did not enroll ninth-grade students, three were missing specific data elements 
(two were missing attendance data and one was missing credit data), and two served transitioning youth. 

** Data were, by definition, missing for excluded students. One suspects that students who were missing data were 
more at-risk than those with data and that their presence in the sample would have lowered achievement means. 
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Variables 



Through hierarchical analyses, we sought to understand the role of NCHS program 
components (see Chapter 4) in generating program outcomes. We paid particular attention to the 
role of schools’ instructional processes and their ways of supporting students’ emotional 
development (e.g., small enrollments, advisories, etc.). In addition, we attempted to control for 
factors such as students’ individual and aggregate characteristics and staff members’ aggregate 
characteristics. 

Student-level criterion variable. Our initial interest was in predicting graduation rates, 
but, at the time of this study, only 10 NCHS schools had graduated a full class of ninth-grade 
entrants. In place of graduation rates, we selected credit accumulation as the criterion variable. 
Credit accumulation is a prerequisite for graduation, and credit data were available for students 
in all NCHS schools. We totaled credit accumulation across two years for students who first 
entered ninth grade in NCHS schools in 2004-05. A description of the dependent variable 
(mean, standard deviation, N) is presented in Exhibit 15. 

Student-level predictor variables. We estimated credit accumulation, in part, based on 
student characteristics. Variables included students’ gender, age, race, special education status, 
immigrant status, English language status, free lunch status, EEA score at entry to high school, 
math score at entry to high school, and attendance in eighth grade. 

One is advised to include only cases with complete data in HEM analyses, and this 
criterion strongly influenced our choice of Eevel 1 variables. Students’ prior reading and math 
scores were both theoretically important but collinear, or highly correlated. We needed to 
choose between these two variables, and we chose math scores to account for students’ prior 
achievement because math scores were available for a higher proportion of NCHS students. 

The final HEM model ultimately included six student-level variables (gender, age, 
special education status, free lunch status, eighth-grade math score, and eighth-grade 
attendance). The variables are presented in Exhibit 15. They were chosen based on their 
theoretical importance, completeness, independence, and apparent relationship with the 
dependent variable. 

School-level predictor variables. Hierarchical models were developed using three sets of 
school-level predictor variables: (1) those describing entrants’ average characteristics, (2) those 
describing the average characteristics of school staff (2005-06), and (3) those describing schools’ 
social and academic conditions and processes. Data describing school conditions and processes 
were gathered in surveys (see Chapter 4) administered to teachers in 2004-05 and 2005-06. 

Preliminary analyses showed that two group-level student variables were associated with 
outcomes (Mean Grade 8 Math Score and Mean Grade 8 Attendance). No staff characteristics 
were associated with outcomes. One process variable was associated with outcomes (Mean 
Score on Quality of Instructional Systems Index). These variables were selected for inclusion in 
HEM analyses based on their theoretical importance, variation, independence, and relationship 
with the dependent variable. 
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Exhibit 15 

Description of Student-Levei and Schooi-Levei Variabies 



Variable Name 


Description 


Criterion 

Credits Over Two 
Years 


A continuous variable indicating credit accumulation over the first two years of high school 
for students entering ninth grade in 2004-05 (M = 23.31, SD = 6. 15, N = 3,871) 


Level 1 Predictors 




Grade 8 Math Score 


A continuous variable indicating score on math test in eighth grade for students entering 
ninth grade in 2004-05 (M = 704.30, SD = 34.20, N = 3,871) 


Grade 8 Attendance 


A continuous variable indicating average daily attendance in eighth grade ter students 
entering ninth grade in 2004-05 (M = 91.92, SD = 6.78, N = 3,871) 


Grade 8 Special 
Education Status 


A dichotomous variable indicating that the student was enrolled in un-graded special 
education program in the year before entry to high school (P = 0.01, N = 3,871) 


Age 


A continuous variable indicating student's age in years at entry to high school for students 
entering ninth grade in 2004-05 (M = 14.75, SD = .56, N = 3,871) 


Gender 


A dichotomous variable indicating student was female (P = 0.54, N = 3,871) 


Free-ZReduced 
Lunch Status 


A dichotomous variable indicating eligibility for lunch assistance at entry to high school for 
students with data (P = 0.93, N = 3,871) 


Level 2 Predictors 




Mean Grade 8 Math 
Score 


A continuous variable indicating for each school its 2005 students’ average math score at 
entry te high school (M = 698.60, SD = 1 1.46, N = 65) 


Mean Grade 8 
Attendance Rate 


A continuous variable indicating for each school its 2005 students’ average attendance rate 
at entry to high school (M = 90.21, SD = 2.29, N = 65) 


Quality of 
Instructional 
Systems in 2004 - 
2006 


A continuous variable indicating for each school the quality of its instructional systems 
based on the composite measure described in Chapter 4, as averaged across school years 
2004-05 and 2005-06 (M = 0.57, SD = 0. 10, N = 65) 



Having framed a basic or preliminary conditional model, we added variables to the model 
from among the possibilities presented in Chapter 4. No other variable increased the explained 
variance. The three Level 2 variables included in preliminary HLM analyses are described in the 
lower panel of Exhibit 15. 



Findings 

In this investigation, HLM was employed in three modalities. The first was an 
unconditional model with no explanatory variables. The second was a conditional model that 
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included explanatory variables of within-group variance (Level 1), and the third was a 
conditional model that included explanatory variables of both within-group variance (Level 1) 
and between-group variance (Level 2). 



Null Model 

The null model seeks to determine the total amount of variability in two-year credit 
accumulation within and between schools and to determine the average amount of credit 
accumulation over two years. As indicated in Exhibit 16, the pooled within-school variance of 
the error term is 31.56, and the variance of the error term between programs is 6.51, yielding a 
Level 1 variance of random effects of 38.07 

The variance of random effects at Level 2 is 6.64. This is the sum of the variance of the 
error term at Level 2 (6.51) and the mean Level 1 error term variance (31.56/246 = .13). 
Procedures for calculating Level 1 and Level 2 variances can be found in Chapter 7 of Snijders 
and Bosker’s Multilevel Analysis (1999).^^ 

The proportion of variance in outcomes between programs (intra-class correlation) is 
estimated at 17.10 percent. The intercept is estimated at 23.18 with a standard error of 0.33. The 
intercept can be interpreted as the expected credit accumulation over two years for a randomly 
selected student drawn from a randomly selected school. 



Conditional Model 1 

Having determined that there was variability among students and school means, the first 
conditional model sought to account for that variability based on six student- level (Level 1) 
variables — three continuous variables (Grade 8 Math Score, Grade 8 Attendance, and Age) and 
three dichotomous variables (Grade 8 Special Education Status, Gender, and Eree-/Reduced 
Eunch Status), as shown in Exhibit 15. 



The concept of “explained variance” or is well-known. Two approaches to explained proportion of variance in 
a two-level model can be defined. The first, and most important, is the proportional reduction of error for predicting 
an individual outcome. The second is the proportional reduction of error for predicting a group mean. 

The Level 1 explained proportion of variance is defined as the proportional reduction in mean square prediction 
error. The most straightforward way to estimate this quotient is to add sigma-squared and tau-squared, as 
determined by the empty model (A). Next, add those parameters as determined by the fitted model (B). Then, 
compute one minus the ratio of the two values or 1 - (B/A). 

The Level 2 explained proportion of variance is the proportional reduction of error for predicting a group mean for a 
randomly drawn Level 2 unit. To estimate this quotient, one must know the n of the usual group size as well as 
sigma-squared and tau-squared, as determined by the empty model and the fitted model. First, determine the value 
of [(sigma-squared/n) + tau-squared] for the null model (A). Next, determine the same value for the fitted model 
(B). The proportional reduction of error for predicting a group mean is estimated at 1 - (B/A). (It is advisable to let 
n reflect the group sizes in the population, rather than the sample group sizes.) 
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Exhibit 16 

Hierarchicai Modeis of Two-Year Credit Accumuiation 







Model 0 Null (empty) model 






Coefficient 


SE 


t 


df 


P 


Fixed effects 












Intercept 


23.18 


0.33 


70.80 


64 


0.00 




Variance Component 


2 

X 


df 


P 


Random effects 












Between-school variance 




6.51 


831.16 


64 


0.00 


Within-school variance 




31.56 












Model 1 


Effects of student-level variables 






Coefficient 


SE 


t 


df 


P 


Fixed effects 












Intercept 


23.47 


0.38 


61.97 


64 


0.00 


Grade 8 Math Score 


0.05 


0.00 


14.30 


3864 


0.00 


Grade 8 Attendance 


0.26 


0.02 


12.83 


3864 


0.00 


Grade 8 Special Education 


-2.08 


0.75 


-2.79 


3864 


0.01 


Age 


-0.68 


0.17 


-3.91 


3864 


0.00 


Free or Reduced Lunch 


-1.01 


0.27 


-3.67 


3864 


0.00 


Female Gender 


1.40 


0.21 


6.75 


3864 


0.00 




Variance Component 


2 

X 


df 


P 


Random effects 












Between-school variance 




4.68 


805.35 


64 


0.00 


Within-school variance 




23.74 












Model 2 


Effects of program-level variables 




Fixed effects 


Coefficient SE 


t 


df 


p 


Intercept 


23.47 


0.38 


61.90 


63 


0.00 


Quality of Instructional 
Systems 


7.00 


2.10 


3.33 


63 


0.00 


Grade 8 Math Score 


0.05 


0.00 


14.24 


3864 


0.00 


Grade 8 Attendanoe 


0.26 


0.02 


12.83 


3864 


0.00 


Grade 8 Special Education 


-2.09 


0.75 


-2.80 


3864 


0.01 


Age 


-0.68 


0.17 


-3.91 


3864 


0.00 


Free-/Reduced Lunch 


-1.02 


0.28 


-3.68 


3864 


0.00 


Female Gender 


1.39 


0.21 


6.70 


3864 


0.00 




Variance Component 


2 

X 


df 


p 


Random effects 












Between-school variance 




4.27 


755.85 


63 


0.00 


Within-school variance 




23.74 
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In the initial model, the continuous covariates were centered around the grand mean and 
set to random. In the final Level 1 model, all covariates were fixed. 

The results of the analysis (with robust standard errors) are presented in the second panel 
of Exhibit 16. All predictor variables show a statistically significant relationship to the outcome. 
Having taken account of Level 1 explanatory variables, the estimated variance of random effects 
at Level 1 was reduced by 25 percent from 38.07 (null model) to 28.42. The variance of random 
effects at Level 2 for Model 1 was reduced from 6.64 to 4.78, or by 28 percent. 



Conditional Model 2 

In the second and final conditional model, efforts were made to account for variability 
based on three continuous Level 2 variables as well as the aforementioned Level 1 variables. 

Two variables described the social contexts of schools based on students’ average entering 
characteristics (Mean Grade 8 Math Score and Mean Grade 8 Attendance Rate) and one 
described school processes (Quality of Instructional Systems in 2004-2006). In the final 
conditional model. Level 1 variables were explained by only one of the three program-level 
variables. Quality of Instructional Systems. 

The estimated variances of the error terms at Level 1 and Level 2 are reported in the final 
panel of Table 16. Note that the Level 1 variance of random effects has been substantially 
reduced from 38.07 in the null model to 28.01 after taking account of Level 1 and Level 2 
explanatory variables. The Level 1 proportion of variance explained by this model is 26.43 
percent. The Level 2 variance of random effects has been reduced from 6.64 in the null model to 
4.37 after taking account of the Level 1 and Level 2 explanatory variables. The Level 2 
proportion of variance explained is 34 percent. 

The expected credit accumulation after two years in an NCHS school was estimated at 
23.48 credits for mainstreamed males, of average age, not eligible for free or reduced price 
lunch, with average prior math scores and average prior attendance. The differentiating effect of 
Grade 8 Math Scores was positive (.05), meaning that students with higher prior performance 
could be expected to accumulate more credits in high school. Lor example, a student whose 
math scores were 20 points higher than average could be expected to acquire 24.48 credits in two 
years, or one additional credit beyond the average student. 

The differentiating effect of prior attendance was also positive (.26). The effects of 
special education status (-2.09), increasing age (-0.68), and free lunch status (-1.02) were, as one 
might expect, all negative. The effect of female gender was positive (1.39), meaning that a 
typical girl earned 1.39 more credits than a typical boy. 

As noted, only one variable describing school processes was associated with variation in 
weighted credit accumulation. The effect of increasing Quality of Instructional Systems was 
positive. Weighted credit accumulation could be expected to increase by seven points (or seven 
credits) as a school moved from a score of 0 or lowest quality instructional systems to a score of 
1.0 or highest quality instructional systems. No school matched the extreme conditions at either 
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end of this hypothetical scale, however. Practically speaking, a typical student’s credit 
accumulation was 1.4 credits higher in a school with a better score on the index (by one standard 
deviation), compared to 1.4 credits lower in a school with a worse score on the index (by one 
standard deviation). 
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8. QUALITATIVE EVIDENCE OF FACTORS 
AFFECTING STUDENT OUTCOMES 



In case studies of six NCHS schools, we aimed to explore three matters in detail: the role 
of school conditions that did not differ sufficiently to be treated as variables in quantitative 
analyses, occasional discontinuities between teacher ratings of instructional processes and 
achievement outcomes, and the relationship between signature NCHS program components 
(such as partnerships) and outcomes. 



Overview of Findings 

Case study data suggested that both principal experience and also conditions that were 
fairly uniform across NCHS schools played an important role in generating positive student 
outcomes. Those conditions included small enrollments, close student-teacher relationships and 
adult mentoring of youth, the extension of student learning outside the regular school setting and 
school day, and the use of data to review progress. 

Case study data also suggested possible explanations for the occasional discontinuities 
between teacher ratings of school conditions and student achievement outcomes. In Case I, 
teacher assessment of school quality was well aligned with student achievement. In Cases 2 and 
3, it was not. In Case 2, teachers made class credits easy to earn, which inflated student 
outcomes beyond the level predicted by the quality of academic processes. In Case 3, the lack of 
alignment between school processes and school outcome data appears to have been a function of 
the variables’ different time horizons. The case data were of the moment and showed an 
inexperienced but improving school staff. Credit accumulation and graduation data represented 
over-time achievements. In that sense, they lagged behind the case data and the survey data. 

Finally, case study data brought the possible impact of data dependencies to our attention. 
In measurement terms (as well as functionally), complex NCHS signature program components 
(like partnerships) were dependent on the systems that created them. This may have interfered 
with efforts to determine in quantitative analyses the specific value they added. 



Methods 

In selecting case study schools, we limited ourselves to those that were relatively well 
established, which were those that opened their doors in the initiative’s first year. From this 
group, we identified six target schools that illustrated a range of student populations, themes, 
partnerships, teacher ratings of instructional processes, and achievement outcomes. 

Within each case study school, we conducted face-to-face interviews with the principal, a 
guidance counselor, and the primary school partner (when there was one). We conducted 
telephone interviews with parents of at least three seniors, and we conducted two focus groups. 
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one with seniors and one with teachers of seniors. The parents, students, and teachers whom we 
interviewed were selected by school personnel. We anticipated, therefore, should our 
impressions be biased, that the direction would be positive. We observed at least one senior- 
level humanities classroom lesson in order to gather some impression, albeit quite limited, of the 
tenor and level of instruction. 

In structured interviews and focus groups, we asked respondents about the dynamics of 
life in their school — how school context, resources (students, staff, partners, discretionary funds, 
and learning time), and processes influenced outputs (programming and instruction) and 
outcomes or results. This sounds tidy, but we were actually looking for the untidy nature of 
interactions, and we asked respondents to consider the tangle of connections that influenced 
results. A specific challenge that we set was to understand what was most puzzling in our 
quantitative data. For example, why did NCHS program components with high theoretical 
utility, themes, advisories, and partnerships, appear to be less influential in quantitative analyses 
than we might have expected? 

The full case study team consisted of seven people, the team leader and three two-person 
pairs of school visitors. Each pair visited two schools and each member of the pair transcribed 
his or her own interviews from those school visits. The lead member of the pair reviewed all 
school-level transcripts, applied a priori codes to the data (derived from interview questions), and 
developed added codes as needed. The lead member next prepared case-level display tables 
organizing data elements by code and wrote memos noting relationships among codes. Tables 
and memos were debriefed within pairs. Over several months, the three pairs met to triangulate 
codes across cases (three meetings), create a framework for displaying the data in multi-case 
matrices (two meetings), and review final case-level memos (one meeting per case). 

In presenting qualitative data, we have chosen to concentrate on three typical NCHS 
schools. The schools are typical in the sense that they served average NCHS students as 
compared to special populations, and they faced challenges that are experienced to one degree or 
another by many NCHS schools. 



Findings 

Case 1 — A Typical NCHS School 

Case 1 was a typical NCHS school in several ways. Its students were average within the 
overall NCHS population. It had average instructional processes, according to teacher ratings, 
and it had a 70-i- percent 2006 graduation rate (also typical within NCHS). 

Respondents attributed the school’s success to several factors: (1) the principal’s 
leadership experience, (2) the community partner’s strengths in after-school programming, GED 
preparation, college advisement, and job placement, and (3) a deepening approach to discipline, 
advisement, and curricula development. 

Here are a few comments regarding the principal’s efforts: 



57 




■ Teacher: “The majority of the teaching staff is eager to make connections with 
students, but [they are] new. The principal is an invaluable resource and leader.” 

■ Principal: “Unlike many others, I wasn’t a new principal. I was experienced in 
working with difficult populations. With a new crop of teachers each year, you 
have to help people learn how to consistently challenge students.” 

Interviewees commented on the partner organization: 

■ Teacher: “The community partner has provided program and financial resources 
to the school we wouldn’t have otherwise had.” 

■ Principal: “Our partner is fully engaged in the advisement program in the school 
and has sought outside funding for extra programming. Our students have 
unrestricted access to our partner’s facilities and support systems. Our partner has 
many things to offer. We just have to be better about accessing those services.” 

Staff members identified some of the school’s challenges: 

■ Teacher: “The admissions process is [awful] and getting into a school you want is 
luck.” 

■ Partner: “Ideally, all of the kids would want to be here. . .but this is not the case.” 

■ Guidance counselor: “A lot of students come in with poor academic skills, and 
we have to do a lot of academic catch up. Some just lack motivation. They don’t 
believe in themselves and have given up.” 

■ Teacher: “Discipline is a time-consuming problem. In the first year, I must have 
logged 40 hours of mediation for one student’s fights.” 

Interviewees commented on the school’s systems for responding to challenges: 

■ Principal: “The school’s theme is how we battle the belief that fighting is okay. 
There’s a formal curriculum in the ninth and tenth grades around self- 
management, but it is also emphasized throughout the day. Still, we need clearer 
consequences for students when they break rules.” 

■ Guidance counselor: “We have a four-year plan to interest students in college. It 
starts in the ninth grade with envisioning the future and why college is important. 
At the end of the twelfth grade, when we know who does not have plans, we help 
students over the summer to complete applications. We also have career fairs to 
highlight post- secondary opportunities that are not directly related to higher 
education or the military.” 
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■ Principal: “We had to find ways of individualizing instruction and advisement. I 
don’t have enough classrooms to run regular classes, let alone separate ELL, 
bilingual, and special education classes. Each discipline has a four-year 
instructional program, but annually teachers make modifications to the curriculum 
based on the profile of the upcoming class of students. One thing that was 
valuable last year, in preparing for the future, was studying our data.” 

■ Principal: “The school has an academic support program for students who are 
over-age and under-credit. The program functions as a one-on-one independent 
study and allows more flexibility than a traditional classroom. The school also 
has an active advisory system that provides students with academic support, 
conflict management skills, and post- secondary exposure. Plus there’s tutoring 
constantly and enrichment.” 



Case 2 — A School with Low Expectations 

This school was typical in only one sense of the word. The student population was more 
or less average for NCHS schools. It was atypical in that teachers judged it to have weak 
instructional processes. And it was atypical in that its graduation rate was over 90 percent, or 
very high. One might ask how these facts could be simultaneously true, and this is what we set 
out to understand through this case analysis. 

Here is a sample of what staff and students said about expectations for student 
achievement and behavior in this school: 

■ Principal: “We have a policy for rigor, but I haven’t seen it. I see a lot of busy 
work. Our teachers have yet to understand what rigor is.” 

■ Teacher: “The school lacks rigorous courses and. . .only offers the minimum state 
standards. Students are apathetic and expect things to be given to them.” 

■ Student: “It’s not challenging. The most challenging coursework occurred during 
the ninth grade.” 

Explanations as to how these conditions emerged centered on staff members’ 
inexperience and their misguided response to student resistance: 

■ Principal: “We didn’t realize the depth and breadth of the academic neglect 
students had experienced. We didn’t realize how hard it was going to be to leave 
the ghetto and the street in the street.” 

■ Principal: “The school was founded to be a nurturing place, but the lenient 
approach to grading and make-up work derailed student motivation. We had, just 
to illustrate, a lot of seniors who didn’t want to take [classes around the theme]. 
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and we let them go home. We have always given them [the students] loopholes 
and incompletes so they could make work up.” 



■ Guidance counselor: “Students could fail a semester and do a project over the 
summer to make up a grade. Grade changes went on like crazy.” 

Despite these policies, students were able to pass Regents exams, in part, because of the 
resources directed at helping them prepare for those exams and other hurdles: 

■ Student: “There are all types of extra help, and teachers stay after school for 
you.” 

■ Teacher: “We have been running academic boot camps to get up the 
statistics . . . .some of the seniors have . . .gone back to night school, and summer 
school to make up for some of their . . .deficiencies.” 

Like most NCHS schools, this one had an investment in counseling and in schoolwide 
themes, but its systems for work on shared professional problems were weak, and staff members 
were uncertain about how to develop unscripted program components: 

■ Guidance counselor: “The school’s advisory system has a great deal of potential, 
but we haven’t worked out a curriculum, and sessions aren’t sufficiently 
standardized or substantial.” 

■ Teacher: “The theme could be a hook, or it could alienate students. As for now, 
we’re not sure what to do with it, and it has very little to do with school 
programming.” 

■ Student: “The theme is absent from the curriculum or in just one class.” 

■ Guidance counselor: “Screening students more rigorously before they entered the 
school might make it possible to integrate the theme without negative 
consequences, but as it stands now, quite a few students just aren’t interested.” 

■ Partner: “Our work is very focused on and relevant to the theme of the school, 
but our resources are seriously under-utilized.” 

Despite the problems, school personnel expressed an impulse toward change and signs of 

hope: 



■ Principal: “We have to come together; we are not just 24 teachers.” 

■ Guidance counselor: “Without buy-in, we’re not going to have a true effect on 
students.” 

■ Teacher: “We need to work together and break down the isolation.” 
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Principal: “There are weekly meetings for teachers to discuss their needs, and the 
school has recently become more data focused.” 



Case 3 — A School with Inexperienced Staff 

This school was typical in two ways. Its students were about average, as were its work 
processes. It was atypical in that the 2006 graduation rate was surprisingly low, under 60 
percent. We wanted to know why results were sub-par. 

According to staff, their abilities to frame expectations and meet students’ instructional 
needs were emerging very slowly: 

■ Teacher: “We almost all had less than five years’ [teaching] experience.” 

■ Teacher: “We’re far more prepared to work with students now than the first 
couple of years.” 

■ Guidance counselor: “My principal wasn’t big on suspensions, but when it 
[behavior problems] became more serious, she had to work on that, and now we 
do in-house suspension when we have to.” 

■ Principal: “Before I was touchy-feely, but as I grow, and the people around me 
grow, expectations and structures are put into place. Now I have zero tolerance 
for do-rags, fighting, cell phones, and hats. It’s about teaching and learning. 
Everyone’s on the same signal. I think kids like knowing the expectations.” 

The school made some headway by incorporating scripted curricula, increasing 
instructional time, and relying on a strong partner: 

■ Principal: “The school now incorporates Ramp Up, 90-minute math periods, and 
90-minute ESL classes to help students learn in a more flexible environment. 
Next year, we plan to run a PM school.” 

■ Teacher: “Teachers offer tutoring on their off-periods. Kids aren’t mandated to 
come, but they often do, and no one’s turned away. They know we want them to 
pass and go on to other things.” 

■ Teacher: “The partner contributes many varieties of learning opportunities for 
students, including a network of academic support.” 

■ Principal: “Our partner is very helpful, especially in getting students to see that 
they are all college bound, but their bureaucracy is [challenging].” 
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Nagging problems remained, however, with regard to development of the curriculum and 
teachers’ professional skills: 

■ Principal: “Only now am I looking at the theme and its instructional relevance. 
The bottom line is academic. The kids must all be college-bound. I’ve focused 
on that.” 

■ Teacher: “Learning opportunities abound for students, but not for teachers.” 

■ Teacher: “Professional development is not helpful. How often can you be taught 
how to make a lesson plan?” 

■ Teacher: “We need more in-house discussions about ways of working in a small 
school.” 

■ Teacher: “We need to talk about the specific needs of our students and our 
school.” 
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9. CONCLUSIONS 



Few school-related research procedures are more debated than ways of capturing 
academic success. Here we have used the percent of students in the Class of 2006 who 
graduated on time as a prime indicator of achievement. This measure has limitations. First, 
schools that accept a high proportion of students with high risk factors have much worse odds of 
achieving success, compared to schools serving students with lower risk factors. Second, 
graduation is an important expectation from schooling, but it is not the equivalent of college 
readiness. 

To cope with the first limitation, we identified the size of the NCHS effect after 
controlling for student characteristics. Looked at in this way, we saw that NCHS schools 
graduated more students on time than larger New York City schools with comparable types of 
youngsters, by about 18 percentage points. Importantly, NCHS schools achieved these results 
without recruiting more experienced teachers or more advantaged students from the school 
system at large. 

To cope with the second limitation regarding expectations for student performance, we 
disaggregated students’ graduation status by the types of diplomas they received. The NCHS 
Class of 2006 had a 78.2 percent four-year graduation rate, compared to the 60.6 percent four- 
year graduation rate in comparison-group schools. Only 36 percent of students in the NCHS 
Class of 2006 received, however, a state-endorsed diploma, compared to 41 percent of students 
in comparison-group schools. 

We conclude that the NCHS intervention was notable with regard to drop-out prevention 
and on-time graduation. Keeping youth in school earning credits and passing exams is a 
significant accomplishment, and it is a basis on which to build deeper accomplishments. 

Through what means did NCHS schools reduce student attrition and improve graduation rates? 
We have answers based on several data sets. 



Civic Explanations 

First among the lessons we extracted from this initiative was that efforts were scalable 
because leaders had opportunities for and embraced joint action. This ignores neither the 
difficulty of collaboration nor the other important ideas about schooling that are at the heart of 
this innovation. It is simply to start with its first causes. 

Financial contributions from the Bill & Melinda Gates Foundation, the Carnegie 
Corporation of New York, and the Open Society Institute quickened a decades-old disposition 
within the school system and among reformers and intermediaries to move aggressively forward 
in creating small high schools. Educators and other civic leaders in New York City were 
experienced with such schools, and labor leaders supported the work. The cohesion among New 
York City stakeholders made it possible for New Visions, the Department of Education, and 
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other stakeholders to super-size the reform. Ultimately, the scale of the work was a distinct 
advantage, as it riveted public and private energies to the project. 



School-Level Explanations 

Our data regarding school-level factors influencing achievement outcomes are strong, 
having been collected over four years using multiple research methods. We’ll start by describing 
easily observed phenomena. 

The simile students regularly used to describe NCHS schools was family-like, and in 
many ways, the reference was apt. The schools were small, and relationships among teachers 
and students were warm, personal, and encouraging. The schools were not, however, 
permissive. Codes of conduct were clear. Students were suspended if they took any part in, let 
alone instigated, aggressive or otherwise dangerous acts. In the best of these schools, serious 
attention was paid through advisories, other forms of counseling, and internships to helping 
students imagine and take steps toward positive futures, not merely stay out of trouble. 

In addition to being friendly and supportive, it was obvious that the schools were places 
of learning. Teachers worked not only on the basics, but also to prepare students for Regents 
exams and to provide experiential learning opportunities. Mixed in with the standard fare of 
instruction, students could find opportunities to travel abroad, double doses of reading and math 
instruction, and individual tutoring. This instructional mix of the traditional, individual, and 
experiential was achieved by extending school time — sometimes the day, the week, and the year. 

Instruction was most successful when collaborative work on curriculum and pedagogy 
was a firm organizational priority. In thriving NCHS schools, principals, master teachers and, at 
times, partners were classroom regulars who directed their energies to expanding teachers’ 
instructional repertoire. Their engagement in instruction and teachers’ reciprocal engagement in 
school decision-making made instructional problem-solving a part of everyday school life. 

These findings are resonant of lessons from successful elementary schools. And they are 
aligned with the reform language of the new 3Rs (rigor, relevance, and relationships), with 
relationships including ideas about the organization of professional work as well as teacher- 
student interactions. 

To the foregoing 3Rs, we would add resources and review. NCHS schools had, through 
the foundations, businesses, and community organizations that helped to create them, the 
wherewithal to enable small-group learning, learning in out-of-school settings, and learning in 
out-of-school time. The schools were also under regular review as a part of the system’s and the 
initiative’s combined focus on accountability. It made sense, in that context, for principals to 
study their data, and we found them doing just that, with help from New Visions. 
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Program Challenges 



One might imagine based on the initiative’s results and the foregoing discussion that 
NCHS schools were all exceptional. This wasn’t the case. NCHS principals and teachers were 
almost always eager novices just learning how to present the basics, to individualize instruction, 
and to provide students with cooperative learning opportunities. 

Growth and sustainability remain real issues for this innovation. NCHS school staff 
members continue to need help in framing expectations for student learning, in wringing value 
from instructional time, and in working collaboratively, both with in-school colleagues and with 
others. 



There are, as well, systemic challenges to be addressed. The high school admissions 
process needs to be simplified. Students transferring out of NCHS schools require additional 
support. And, based on the performance of the NCHS schools, the process for system renewal 
through school closings and new- school creation needs to be sustained. 

Taken together, the final two points suggest a particular challenge. How can the school 
system meet the needs of students who transfer out of NCHS schools? These schools are among 
the best options the system has to offer, especially for keeping educationally at-risk students in 
school. Students who transfer out of NCHS schools to other city schools are very likely to drop 
out later. New York City needs more convenient, appealing options for this population of 
youngsters. 



Research Support Going Forward 

Focusing on results has powerful effects, but a few are subversive. To keep the focus on 
best practices and off quick fixes, education policymakers will need to make full use of the 
school-process data newly available as part of New York City’s performance feedback system. 

There is no more efficient and reliable way of understanding schools than by asking 
teachers about their work lives. Over time, we hope the new annual teacher survey will provide 
the system with both school-specific and citywide feedback on the quality of critical school 
systems, including those for: recruiting teachers; setting academic and social expectations for 
students; creating, adapting, and implementing new curricula; assessing and developing staff 
capacities; and vetting and engaging in partnerships with external experts and organizations. 
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Appendix A 

Descriptive Statistics on Survey Data Presented in Chapter 4 



Item 1 : Teacher Expectations 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.90 


.01 


974 


.89 


.10 


1541 


.89 


.01 


2338 


2 








.87 


.01 


1643 


.86 


.01 


2487 


3 














.90 


.01 


3188 


ALL 


.90 


.01 


974 


.88 


.01 


3184 


.89 


.00 


8013 



Index 1 : Regents Alignment 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.46 


.05 


64 


.51 


.04 


111 


.47 


.03 


185 


.56 


.03 


213 


2 








.66 


.04 


90 


.64 


.03 


203 


.68 


.02 


267 


3 














.59 


.03 


217 


.63 


.02 


430 


ALL 


.46 


.05 


64 


.58 


.03 


201 


.57 


.02 


605 


.63 


.01 


910 



Index 2: Student Self-Management 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.61 


.04 


70 


.60 


.03 


118 


.54 


.02 


198 


.45 


.02 


226 


2 








.73 


.03 


93 


.48 


.02 


245 


.43 


.02 


274 


3 














.61 


.02 


244 


.48 


.02 


467 


ALL 


.61 


.04 


70 


.65 


.02 


211 


.55 


.01 


657 


.46 


.01 


967 



68 
















Index 3: Teacher-Student Relationships 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.73 


.01 


937 


.77 


.01 


1480 


.76 


.01 


2250 


2 








.71 


.01 


1541 


.68 


.01 


2379 


3 














.75 


.01 


3013 


ALL 


.73 


.01 


937 


.74 


.01 


3021 


.73 


.00 


7642 



Item 2: Advisories 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.74 


.01 


69 


.70 


.04 


117 


.77 


.03 


198 


.66 


.03 


223 


2 








.72 


.05 


93 


.63 


.03 


215 


.50 


.03 


275 


3 














.78 


.03 


240 


.61 


.02 


469 


ALL 


.74 


.01 


69 


.71 


.03 


210 


.73 


.02 


653 


.59 


.02 


967 



Item 3: Number of Students Taught 



School Group 


Data Collection Year | 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


70.99 


2.38 


71 


78.38 


3.63 


100 


81.91 


3.45 


158 


81.71 


3.35 


171 


2 








83.39 


3.71 


79 


83.68 


3.09 


175 


91.16 


3.15 


203 


3 














80.15 


2.18 


192 


88.70 


1.88 


372 


ALL 


70.99 


2.38 


71 


80.59 


2.61 


179 


81.86 


1.66 


525 


87.77 


1.49 


746 



69 














Index 4: Educational Focus 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.75 


.03 


66 


.74 


.02 


116 


.71 


.02 


195 


.65 


.02 


219 


2 








.76 


.02 


93 


.74 


.02 


211 


.65 


.02 


272 


3 














.75 


.02 


235 


.69 


.01 


460 


ALL 


.75 


.03 


66 


.75 


.02 


209 


.73 


.01 


641 


.67 


.01 


951 



Index 5: Principal Leadership 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.69 


.05 


66 


.78 


.03 


115 


.74 


.02 


189 


.64 


.02 


219 


2 








.84 


.03 


88 


.79 


.02 


212 


.73 


.02 


269 


3 














.84 


.02 


236 


.77 


.01 


457 


ALL 


.69 


.05 


66 


.81 


.02 


203 


.80 


.01 


637 


.73 


.01 


945 



Index 6: Professional-Development Quality 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.44 


.04 


65 


.45 


.03 


113 


.46 


.03 


189 


.35 


.02 


218 


2 








.44 


.04 


91 


.36 


.02 


210 


.32 


.02 


266 


3 














.46 


.02 


238 


.35 


.02 


455 


ALL 


.44 


.04 


65 


.44 


.02 


204 


.43 


.01 


637 


.34 


.01 


939 



70 













Index 7: Professional-Development Quantity 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.40 


.04 


62 


.42 


.03 


107 


E| 




191 


.38 


.02 


222 


2 








.43 


.04 


86 




■1 


212 


.39 


.02 


269 


3 
















.02 


235 


.40 


.02 


450 


ALL 


.40 


.04 


62 


.42 


.02 


193 




.01 


638 


.39 


.01 


941 



Index 8: Teacher Influence 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.66 


.04 


70 


.58 


.03 


118 


.52 


.02 


193 


.45 


.02 


223 


2 








.64 


.03 


92 


.43 


.02 


212 


.41 


.02 


273 


3 














.64 


.02 


237 


.52 


.01 


458 


ALL 


.66 


.04 


70 


.61 


.02 


210 


.54 


.01 


642 


.47 


.01 


954 



Index 9: Teacher Collaboration 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.75 


.03 


71 


.77 


.02 


114 


.78 


.02 


199 


.71 


.02 


223 


2 








.75 


.03 


91 


.70 


.02 


213 


.63 


.02 


274 


3 














.76 


.02 


237 


.71 


.01 


460 


ALL 


.75 


.03 


71 


.76 


.02 


205 


.75 


.01 


649 


.69 


.01 


957 



71 















Index 10: Meaningful Assessments 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.70 


.01 


962 


.69 


.01 


1479 


.66 


.01 


2248 


2 








.65 


.01 


1563 


.64 


.01 


2353 


3 














.71 


.01 


3025 


ALL 


.70 


.01 


962 


.67 


.01 


3042 


.68 


.00 


7626 



Index 11: Community Partners’ Roie 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.48 


.09 


8 


.59 


.09 


11 


.51 


.09 


12 


.45 


.10 


12 


2 








.43 


.08 


15 


.40 


.06 


16 


.36 


.06 


14 


3 














.46 


.05 


35 


.43 


.04 


36 


ALL 


.48 


.09 


8 


.50 


.06 


26 


.45 


.03 


63 


.42 


.03 


62 



Item 4: Parental Involvement 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.44 


.06 


71 


.35 


.04 


115 


.38 


.03 


195 


.36 


.03 


222 


2 








.73 


.05 


96 


.40 


.03 


213 


.36 


.03 


270 


3 














.48 


.03 


245 


.36 


.02 


461 


ALL 


.44 


.06 


71 


.52 


.03 


211 


.43 


.02 


653 


.36 


.02 


953 



72 














Index 12: Student Engagement 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.38 


.01 


906 


.40 


.01 


1479 


.37 


.01 


2236 


2 








.42 


.01 


1527 


.35 


.01 


2358 


3 














.40 


.01 


3015 


ALL 


.38 


.01 


906 


.41 


.01 


3006 


.37 


.00 


7609 



Index 13: Learning Opportunities 



School Group 


Data Collection Year 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


1 


.87 


.07 


10 


.86 


.05 


11 


2 


.86 


.06 


15 


.86 


.06 


14 


3 


.73 


.05 


35 


.85 


.04 


36 


ALL 


.79 


.03 


60 


.86 


.03 


61 



Index 14: Computer Use 



School Group 


Data Collection Year | 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.62 


.05 


65 


.62 


.04 


109 


.69 


.03 


191 


.62 


.03 


220 


2 








.67 


.04 


87 


.60 


.03 


201 


.49 


.03 


267 


3 














.60 


.03 


219 


.53 


.02 


444 


ALL 


.62 


.05 


65 


.64 


.03 


196 


.63 


.02 


611 


.54 


.01 


931 



73 















Item 5: Availability of Technoiogy 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.41 


.06 


66 


.52 


.05 


111 


.49 


.04 


190 


.45 


.03 


215 


2 








.45 


.05 


91 


.45 


.04 


199 


.37 


.03 


262 


3 














.45 


.03 


223 


.48 


.02 


445 


ALL 


.41 


.06 


66 


.49 


.04 


202 


.46 


.02 


612 


.44 


.02 


922 



Item 6: Instructional Materials 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.71 


.06 


69 


.81 


.04 


113 


.82 


.03 


195 


.68 


.03 


221 


2 








.82 


.04 


93 


.72 


.03 


213 


.69 


.03 


277 


3 














.73 


.03 


237 


.65 


.02 


460 


ALL 


.71 


.06 


69 


.81 


.03 


206 


.75 


.02 


645 


.67 


.02 


958 



Item 7: Teacher Degree 



School Group 


Data Collection Year 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.74 


.04 


115 


.66 


.03 


185 


.71 


.03 


195 


2 


.67 


.05 


89 


.72 


.03 


201 


.75 


.03 


235 


3 








.70 


.03 


223 


.67 


.02 


421 


ALL 


.71 


.03 


204 


.69 


.02 


609 


.70 


.02 


851 



74 













Item 8: Teacher Certification 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.30 


.06 


67 


.24 


.04 


111 


.28 


.03 


180 


.29 


.03 


228 


2 








.26 


.05 


91 


.33 


.03 


195 


.27 


.03 


285 


3 














.29 


.03 


215 


.25 


.02 


483 


ALL 


.30 


.06 


67 


.25 


.03 


202 


.30 


.02 


590 


.27 


.01 


996 



Item 9: Facilities 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.24 


.05 


70 


.38 


.05 


116 


.29 


.03 


199 


.31 


.03 


225 


2 








.39 


.05 


96 


.22 


.03 


218 


.30 


.03 


279 


3 














.44 


.03 


244 


.42 


.02 


472 


ALL 


.24 


.05 


70 


.38 


.03 


212 


.32 


.02 


661 


.36 


.02 


976 



Item 10: Host School 



School Group 


Data Collection Year | 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.59 


.06 


68 


.69 


.04 


109 


.67 


.03 


188 


.61 


.03 


214 


2 








.63 


.05 


95 


.60 


.03 


213 


.66 


.03 


264 


3 














.70 


.03 


240 


.68 


.02 


456 


ALL 


.59 


.06 


68 


.66 


.03 


204 


.66 


.02 


641 


.66 


.02 


934 



75 












Index 15: Quality of Instructional Systems 



School Group 


Data Collection Year 


2002-03 


2003-04 


2004-05 


2005-06 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


M 


SE 


N 


1 


.60 


.03 


67 


.61 


.02 


113 


.59 


.01 


188 


.54 


.01 


222 


2 








.64 


.02 


91 


.58 


.01 


209 


.55 


.01 


264 


3 














.64 


.01 


233 


.58 


.01 


448 


ALL 


.60 


.03 


67 


.62 


.01 


204 


.61 


.01 


630 


.56 


.01 


934 



76 








Appendix B 

Means Comparisons: Entering Characteristics of Ciass of 2006 Students 
in NCHS and Comparison-Group Schoois 





t-test for Equality of 
Means 


Variables 


Group 

(NCHS=1) 


N 


Mean 


Std. 

Deviation 


Std. Error 
Mean 


Mean 

Difference 


Sig. 

(2-tailed)* 


Reading Score 


.00 

1.00 


4447 

666 


686.56 

685.52 


22.39 

18.03 


.34 

.70 


1.04 


.180 


Math Score 


.00 

1.00 


4692 

681 


696.60 

693.49 


32.43 

28.14 


.47 

1.08 


3.11 


.008* 


Black 


.00 

1.00 


5802 

855 


.30 

.35 


.46 

.48 


.01 

.02 


-.05 


.002* 


Hispanic 


.00 

1.00 


5802 

855 


.54 

.52 


.50 

.50 


.01 

.02 


.03 


.151 


Recent Immigrant 


.00 

1.00 


5802 

855 


.09 

.12 


.29 

.33 


.00 

.01 


-.03 


.008* 


ELL Services Eligibility 


.00 

1.00 


5802 

855 


.13 

.12 


.33 

.32 


.00 

.01 


.01 


.392 


Lunch Eligibility 


.00 

1.00 


4763 

830 


.92 

.91 


.28 

.29 


.00 

.01 


.01 


.339 


Gender 


.00 

1.00 


5802 

855 


.55 

.56 


.50 

.50 


.01 

.02 


-.01 


.637 


Special Education Eligibility 


.00 

1.00 


5802 

855 


.05 

.05 


.21 

.22 


.00 

.01 


.00 


.628 


Age 


.00 

1.00 


5802 

855 


14.84 

14.85 


.68 

.74 


.01 

.03 


-.01 


.758 



* Indicates statistical significance. 
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